Detecting and Using Body Tissue Electrical Signals

ABSTRACT

Systems and methods for gesture control are disclosed. In some embodiments, a system may include a plurality of electrode pairs, a motion sensor, a controller, and a classifier. The system may be configured to: enter a monitoring state in which the system is configured to receive data; receive a first set of data; determine that the first set of data does not satisfy one or more action criteria; return to the monitoring state without transmitting the first set of data to the classifier; receive a second set of data; determine that the second set of data satisfies the one or more action criteria; transmit the second set of data to the classifier; using the classifier, analyze the second set of data to generate an interpreted output indicating a gesture performed by the person; and based on the interpreted output, generate a machine instruction.

This application is a continuation application of U.S. nonprovisionalapplication Ser. No. 16/104,273, filed on Aug. 17, 2018, which is acontinuation of U.S. nonprovisional application Ser. No. 15/826,131,filed on Nov. 29, 2017 and issued as U.S. Pat. No. 10,070,799, which isentitled to the benefit of the filing date of U.S. provisionalapplication Ser. No. 62/429,334, filed on Dec. 2, 2016, and U.S.provisional application Ser. No. 62/566,674, filed on Oct. 2, 2017, theentire contents of all of which are incorporated here by reference.

BACKGROUND

This description relates to detecting and using body tissue electricalsignals.

The peripheral nerves in a human body, for example, produce andpropagate electrical signals (sometimes called impulses) to do variousjobs. Efferent nerves, for example, propagate signals from the centralnervous system to muscles of the extremities to cause contraction orrelaxation of the muscles. (For simplicity, we sometimes use the term“contraction” to refer to both contraction or relaxation.) Afferentnerves propagate signals in the other direction, from sensory organsincluding the skin to the central nervous system. One way to detectnerve electrical signals (impulse activity) is by the invasive techniqueof inserting an electrode through the skin and into the nerve; the nerveelectrical signals can then be detected at the exposed end of theelectrode.

Contraction or relaxation of muscles is caused by neurons at locationsin the vicinity of, for example, muscles that operate the fingers. Toactivate a particular muscle (e.g., to cause it to contract), the braincan send a signal to a particular selected nerve having a bundle ofneurons to cause the neurons to produce electrical signals. Theseelectrical signals are based on the chemical process of interactionbetween neurons and between neurons and effector cells through thechemical messengers or neurotransmitters that are released. In the caseof skeletal muscle cells, activated motor neurons transmit theneurotransmitter acetylcholine to the synapses of the skeletal musclecells allowing the ion channels in that cell to open. This changes themembrane potential of the cell resulting in the sequence of events thatcauses an action potential and then a contraction of the muscle.

By sending coordinated electrical signals through selected nerves tocorresponding muscles associated with a particular finger, for example,the brain can cause the finger to produce a wide variety of actions suchas touching the palm. Thus, the brain can send signals through nerves tomuscles in various parts of the body to cause parts of the body to moveaccording to gestures intended by the brain, such as raising the indexand middle finger of a hand in a “V”. The gestures in turn can representintent, for example, the intent to congratulate a teammate on a victory.

Other tissues of the body, such as brain cells, interneurons, and bones(which exhibit piezoelectric properties), also can produce electricalsignals. Electrical signals produced by some tissues of the body aresometimes called action potentials.

SUMMARY

In general, in an aspect, bio-potentials (e.g., any electrical potentialgenerated biologically that can be) are sensed on the skin of a subject.The bio-potentials include muscle bio-potentials and nervebio-potentials. Skin sensors are positioned to enable the sensingcircuitry to emphasize the nerve bio-potentials and deemphasize themuscle bio-potentials in processed bio-potential signals generated bythe sensing circuitry. A machine learning component identifies controlsequences or tracks body motions based on the processed bio-potentialsignals.

In general, in an aspect, electrical potentials are detected on thesurface of the skin of a wrist or ankle (for example, or other bodypart), and the detected electrical potentials are processed to providedata values representing electrical signals occurring at nerves in thewrist or ankle.

Implementations may include any one or a combination of two or more ofthe following features. A structure supports the detectors and circuitryon the wrist. The detectors include electrodes. Alignment features alignthe structure relative to an upper surface and an outer edge of thewrist to position the detectors in contact with skin at the wrist orankle and in the vicinity of particular internal tissues of the wrist.The alignment features are configured to position the detectors incontact with the top of the wrist or a particular surface of the ankle.The alignment features are configured to position the detectors incontact with the bottom of the wrist. The alignment features arearranged in a “J” and include a generally flat section to engage theupper surface of the wrist and a convex section to engage the outer sideof the wrist. The alignment features are part of a single integratedrigid piece. The alignment features include flexible features. Thealignment features include rigid features. The detectors are exposed atone or more internal surfaces. The detectors include pairs of detectors,one of the detectors of each pair being closer to the fingers or toesthan the other detector of the pair. There can be at least twodetectors, at least six detectors, at least twelve detectors, or atleast one hundred detectors. At least one of the detectors is positionedto contact skin on the outer side of the wrist or ankle. At least one ofthe detectors is positioned to contact skin on the upper surface of thewrist. Each of the detectors includes a dry contact surface that isassociated with a conductive solid, gel, foam, or film. Each of thedetectors can be embedded in the alignment features as separatecomponents, combined within a material used to form the alignmentfeatures, or can be fibers within a fabric. There is a resilientmounting for at least one of the detectors. The resilient mounting urgesthe detector to project from an inner surface. A band holds thestructure on the wrist or ankle. The alignment features can interfacewith wearable tattoos or skin-based circuits. The alignment featuresinclude an edge to bear against a projection of the ulna (in the wristexample). The circuitry includes electronic components. The electroniccomponents include a notch filter having an input coupled to at leastone of the detectors. The electronic components include (e.g., aprogrammable hardware) bandpass filter associated with at least one ofthe detectors. The electronic components include an amplifier and ananalog-to-digital converter. The electronic components include aninertial measurement unit. The electronic components include a wirelesscommunication component. The electronic components include a processorto execute stored instructions to process digital signals derived fromthe detectors. The stored instructions are to cause the processor toproduce data values. The stored instructions are to cause the processorto apply a process to extract features from data values and producecharacterizations about gestures or intent or muscle contractions. Theprocess includes a classifier. The stored instructions are to cause theprocessor also to use information from one or more other data sources.One of the other sources includes an inertial measurement unit. Thestored instructions are to cause the processor to control a selection ofgroupings of two or more detectors from which signals will be acquiredand processed as channels. One of the detectors in one of the groupingsincludes an electrode external to the wrist device. One of the channelsis associated with three of the electrodes. The electronic componentsinclude a user interface. The user interface includes a display. Theuser interface includes one or more button controlled switches and amulti-color intensity controllable LED. The user interface includeshaptic functions. There is a port for connection of an electricalconductor between the port and a detector configured to be adhered tothe skin of the user. Functions are provided other than detectingelectrical signals on the skin. The functional components include a timepiece. A housing for the structure has the alignment features, aprocessor, a battery, a wireless communication module, and a userinterface. There is a USB port for coupling a processor to an externaldigital device. There is a wireless communicator for coupling aprocessor to an external digital device.

In general, in an aspect, on the skin of a subject, electricalpotentials are detected that are characteristic of electrical activityof neurological tissue and electrical potentials are also detected thatare characteristic of electrical activity of non-neurological tissue inthe vicinity of the detector. There is a processor, and digital storagecontaining instructions to be executed by the processor to cause theprocessor to produce data values representative of the electricalactivity of the neurological tissue based on the detected electricalpotentials.

Implementations may include any one or a combination of two or more ofthe following features. The instructions are to cause the processor tointerpret the data values representative of the electrical activity ofthe neurological tissue as being associated with contractions ofparticular muscles. The instructions are to cause the processor toidentify one or more gestures corresponding to the detected electricalpotentials. The instructions are to cause the processor to identifyintent corresponding to the detected electrical potentials. Theinstructions are to cause the processor to apply a process to featuresof the data to infer actions of particular muscles or gestures or intentbased on known relationships of the features to the actions or gesturesor intent. The process includes a machine learning process. The processincludes a classifier process. The process includes a thresholdingprocess. The electrical potentials are detected on the skin of the wristof the subject. The detector includes electrodes to contact the skin ofthe subject. The detector includes electrodes to be used in pairs todetect differential electrical potentials on the skin. The detectorincludes a predefined pattern of electrodes to contact the skin of thesubject, the instructions are to cause the processor to interpret thedata as being associated with one or more gestures, muscle actions, orintent, and the system includes stored information representative ofknown relationships between electrical potentials detected by thepredefined pattern of electrodes and corresponding gestures, muscleactions, or intent. There are one or more signal processing componentsto receive the detected electrical potentials and to produce processedelectrical signals that are representative of, and have a bettersignal-to-noise ratio than, the detected electrical potentials of theneurological tissue. A spectral filter produces the processed electricalsignals by spectral filtering the detected electrical potentials basedon known spectral characteristics of electrical activity of neurologicaltissue. A notch filter produces the processed electrical signals byfiltering out ambient electrical noise, based on known spectralcharacteristics, to obtain the biopotential signal. A temporal filterremoves or attenuates frequencies in the biopotential signal. This helpsto improve the signal-to-noise ratio of the signal. A spatial filter candetermine each detector's amplitude based on the distance from eachother. An analytical component produces the processed electrical signalsby analyzing successive times of occurrence or changing polarity ofelectrical potentials at two locations on the skin and identifyingelectrical potentials characteristic of electrical activity ofneurological tissue based on the relationship of the times of occurrenceor the polarities. One or more electrical potentials characteristic ofelectrical activity of muscle tissue are detected on the skin of thesubject, and the detected muscle tissue electrical potentials are usedfor noise cancellation with respect to the electrical potentialsdetected for the neurological tissue. An impedance matching componentproduces the processed electrical signals by matching the impedance ofthe detector to the impedance of the skin. The impedance is matchedmanually or dynamically. The impedance matching component controls aprogrammable resistor. The impedance matching component receivesreal-time measurements corresponding to the impedance on the skin of thesubject.

The detector includes an array of electrical potential detectors and theinstructions are to cause the processor to select a set of two or moreof the detectors, the data being produced based on the electricalpotentials of the selected set of detectors. There is a communicationdevice. The instructions are to cause the processor to communicate thedata to a therapy system through the communication device. Theinstructions are to cause the processor to communicate the data to adiagnostic system through the communication device. The instructions areto cause the processor to communicate the data to a control systemthrough the communication device. The instructions are to cause theprocessor to communicate the data to a feedback system through thecommunication device. The instructions are to cause the processor tocommunicate the data to a research system through the communicationdevice. The instructions are to cause the processor to communicate thedata to a performance measurement system through the communicationdevice. The instructions are to cause the processor to perform acalibration operation. The instructions are to cause the processor tocommunicate the data to an augmented reality or virtual reality systemthrough the communication device.

In general, in an aspect, a device is controlled using electromyographysignals measured from a skin surface using one or more surfaceelectromyography sensors and a sensor hub.

In general, in an aspect, gesture or intent information is determinedbased on nerve electrical activity and is provided to an application.

Implementations may include any one or a combination of two or more ofthe following features. The instructions to be executed by the processorare to cause the processor to receive the gesture or intent informationwirelessly from a device having electrical potential detectors on theskin of a user. The instructions to be executed by the processor are tocause the processor to receive other information not representingelectrical potentials detected on the skin of a user, and to generatethe gesture or intent information based also on the other information.The other information includes inertial information from an inertialmeasurement unit. The processor is part of the user device and theinstructions to be executed by the processor are to cause the processorto provide the gesture or intent information to two or more otherapplications running on the user device. The gesture or intentinformation is received wirelessly from two or more such devices and isexpressed in a device-independent format. The gesture or intentinformation is associated with a succession of times based on nerveelectrical activity occurring at the succession of times. The gesture orintent is based on motion of one or more fingers and the gesture orintent information is provided to the application for use with a physicsmodel of finger motion to control motion of fingers on a user interface.The instructions to be executed by the processor are to cause hapticfeedback to a subject who is the source of the gesture or intent.

In general, in an aspect, on the skin of a subject, electricalpotentials are detected that are characteristic of electrical activityof tissues including electrical potentials characteristic of each ofnerve tissue and muscle tissue. The instructions to be executed by theprocessor are to cause the processor to produce data representative ofthe separate electrical activity of each of the tissues based on thedetected electrical potentials.

Implementations may include any one or a combination of two or more ofthe following features. The instructions are to cause the processor tointerpret the data representative of the electrical activity of therespective tissues as being associated with actions of particularmuscles. The instructions are to cause the processor to infer one ormore gestures or intent corresponding to the detected electricalpotentials for respective tissues. The instructions are to cause theprocessor to apply a process to features of the data to infer actions ofparticular muscles or gestures or intent based on known relationships ofthe features to the actions, intent, or gestures. The process includes amachine learning process. The process includes a classifier process. Theelectrical potentials are detected on the skin of the subject. Theelectrical potentials are detected on the skin of the wrist. The sensorincludes electrodes to contact the skin of the subject. The sensorincludes electrodes to be used in pairs to detect differentialelectrical potentials on the skin. The sensor includes a predefinedpattern of electrodes to contact the skin of the subject, theinstructions are to cause the processor to interpret the data as beingassociated with one or more gestures, and the system includes storedinformation representative of known relationships between electricalpotentials detected by the predefined pattern of electrodes andcorresponding gestures. One or more signal processing components receivethe detected electrical potentials and produce processed electricalsignals that are representative of, and have a better signal-to-noiseratio than, the detected electrical potentials. The producing of theprocessed electrical signals includes spectral filtering the detectedelectrical potentials based on known spectral characteristics ofelectrical activity of respective tissues. The producing of theprocessed electrical signals includes notch filtering the detectedelectrical potentials based on known spectral characteristics of ambientnoise. The producing of the processed electrical signals includesmatching the impedance of the sensor to the impedance of the skin. Theimpedance is matched dynamically. The impedance matching includescontrolling a programmable resistor. The impedance matching includesreceiving real-time measurements corresponding to the impedance on theskin of the subject. The sensor includes an array of electricalpotential detectors and the instructions are to cause the processor toselect a set of two or more of the detectors, the data being producedbased on the electrical potentials of the selected set of detectors. Theinstructions are to cause the processor to select a series of sets ofdetectors, the data being produced based on the electrical potentialsform each of the sets of detectors. There is a communication device. Theinstructions are to cause the processor to communicate the data to atherapy system, a diagnostic system, a control system, a feedbacksystem, a research system, or a performance measurement system. Theinstructions are to cause the processor to interpret the datarepresentative of the electrical activity of the neurological tissue asbeing associated with intentions of the subject. The instructions are tocause the processor to perform a calibration operation. The instructionsare to cause the processor to communicate the data to an augmentedreality or virtual reality system.

In general, in an aspect, activation of movement of an appendage of avertebrate subject is monitored. A set of at least two electrodescontacts a skin surface of the vertebrate in a location proximate to aneuromuscular bundle associated with the appendage. A matrix mounts theset of electrodes in position in the location. A signal processing unithas an input coupled to the electrodes and an output. The unit includes:an amplifier system coupled to the unit's input and having an output; afilter system that passes neuronal signals and attenuates other signals;and a postprocessor coupled to the output.

Implementations may include any one or a combination of two or more ofthe following features. The postprocessor is a metering device toprovide a measure of a level of neuronal activity associated with theactivation of movement. The postprocessor includes a threshold processorto provide an output only when a level of neuronal activity exceeds athreshold.

In general, in an aspect, and apparatus includes a processor and adigital storage containing instructions executable by the processor to(a) determine gesture or intent information of a person based on nerveelectrical activity detected on skin of the person and (b) provide thegesture or intent information to an application, the gesture or intentbeing about motion of the hand or wrist or finger or a combination ofthem of the person.

Implementations may include one or a combination of two or more of thefollowing features. The motion includes rolling, tilting, or yawing ofthe hand or the wrist or a combination of the hand in the wrist. Themotion includes rotating or twisting the hand and wrist. The gesture orintent information includes at least one of direction, orientation, rateof motion, or magnitude. The motion corresponds to manipulation of asphere. The gesture or intent information is associated with a motion ofa displayed element of a displayed user interface of the application.The displayed element includes a sphere or an element analogous to asphere. The displayed element is proportional to the motion of thegesture or intent. The motion of the displayed element is notnecessarily proportional to the motion of the gesture or intent. Themotion of the displayed element is translated to motion of the displayedelement according to a mapping. The magnitude of the motion of thedisplayed element is a multiple of the magnitude of the motion of thegesture or intent. The magnitude of the motion of the displayed elementis based on a magnitude of acceleration of the motion of the gesture orintent.

Other aspects, implementations, features, and advantages, andcombinations of them, can be expressed as methods, apparatus, systems,components, means and steps for performing the function, programproducts, software, business methods, and in other ways.

Other aspects, features, implementations, and advantages will becomeapparent from the following description, and from the claims.

DESCRIPTION

FIG. 1A is a diagram of an example sensor hub in communication with sEMGsensors and interfacing with a client device;

FIG. 1B is a diagram of a client device and an EMG controller.

FIG. 2 is a flowchart for an example method of monitoring sEMG sensors;

FIG. 3A, FIG. 3B, and FIG. 3C are screen images for an exampleimplementation of a sensor monitor and interface;

FIG. 4 is a flowchart for an example method of selecting a matrix fieldusing sEMG sensors;

FIG. 5 is a flowchart for an example method of operating a pointer usingsEMG sensors;

FIG. 6 is a flowchart for an example method of selecting a menu optionusing sEMG sensors;

FIG. 7 is a flowchart for an example method of differentiating multiplesEMG sensors; and

FIG. 8 is a block diagram of a computing system suitable for use in thevarious implementations described.

FIG. 9 is a schematic block diagram.

FIG. 10 is a perspective drawing of a wrist device.

FIG. 11 is an exploded perspective drawing of a wrist device.

FIGS. 12, 13, and 14 are perspective, side, and perspective viewsrespectively of an electrode snap arrangement.

FIG. 15 is a top view of part of a wrist device.

FIG. 16 is a block diagram of signal processing circuitry.

FIG. 17 is a block diagram of a processing device.

FIG. 18 is a schematic diagram of control of a displayed user interface.

FIG. 19 is a schematic block diagram.

OVERVIEW

Here we describe, among other things, technologies that can detecttissue electrical signals (in particular, nerve electrical signals)non-invasively, accurately, and precisely, and use information about theelectrical signals for a wide variety of purposes including diagnosis,therapy, control, inference of gestures or other indications of intent,bio-feedback, interaction, analysis, and study, to name a few. Becausethe information about the electrical signals can be more accurate andprecise than other techniques, the purposes for which they are used canbe more effective, faster, simpler, subtler, and more productive, amongother things. New applications not currently possible can also beimplemented.

We use the term “nerve” broadly to include, for example, any part of abody that carries signals, particularly electrical signals (e.g.,electrochemical signals) between parts of the body. Nerves includeefferent nerves that are part of the peripheral nervous system and causemuscles to, for example, contract or relax, and we sometimes use theterm “nerve” to refer to efferent nerves. Nerves also include afferentnerves that carry signals as part of the sensory nervous system, and wesometimes use the term “nerve” to refer also to afferent nerves. A nervetypically comprises a bundle of fascicles each of which comprises abundle of neurons.

We use the term “electrical signal” broadly to include, for example, anyevent, occurrence, or phenomenon that can be detected electrically, suchas electrical potential, resistance, capacitance, inductance, otherimpedance, or current. We sometimes use the term “electrical signal” torefer to such an event, occurrence, or phenomenon that occurs at a placein the body, can propagate through tissues of the body, and may bedetected or sensed at a (e.g., nearby) location on the skin. Forsimplicity, we sometimes use the term “electrical potential” broadly toinclude, for example, electrical potential, resistance, capacitance,inductance, other impedance, or current.

As an electrical signal propagates along a given nerve, it can bedetected at a succession of locations along the length of the nerve andits direction and speed of propagation can be determined by detectingthe times of appearance and the polarities of the electrical signal attwo (e.g., closely) spaced apart locations and determining how much timehas elapsed between those times.

In our discussion, we often use as an example the detection and use ofnerve electrical signals that appear in the wrist. The technologies thatwe describe here also can be useful for detecting and using nerveelectrical signals or other tissue electrical signals at any other partof the body, especially the extremities.

Although the technologies that we describe here apply to any part of theperipheral nervous system generally, they are particularly useful forthe somatic or voluntary nervous system which effects voluntary controlof skeletal muscles.

In the physiology of the human body, specific nerves or parts of themmay be embedded in or closely associated with muscles, bones, and othertissues. Electrical signals in the body may occur not only in nerves,but also in muscles, bones, and other tissues (all of which we sometimesrefer to as “signal sources”).

Many parts of the human body contain combinations of these signalsources intertwined, overlaid, or otherwise near to one another. Anelectrical signal occurring at one of the signal sources typicallypropagates in all directions and is attenuated and dispersed to degreesdetermined by the distances it propagates and by the attenuationproperties of other tissues through which it propagates. Eventually thepropagated, attenuated versions of the original electrical signals thatoccurred at the signal sources reach the surface of the skin where theyform detectable electrical potentials, for example. Because electricalsignals from two or more of the signal sources (say a muscle and anearby nerve) can occur at roughly the same time, the electricalpotential at a given location on the skin at a given time typicallyrepresents a complex aggregation of electrical potentials that resultfrom electrical signals occurring at multiple signal sources. Theelectrical potentials that appear at a given time and at successivetimes at an array of locations in a given region on the surface of theskin, say at the wrist, therefore are indicative of specific electricalsignals occurring at each of the signal sources in the vicinity of thatregion within the body at that given time and sequence of times.

By detecting electrical potentials at such an array of locations in thegiven region on the skin and processing information representing thoseelectrical potentials, the technologies that we describe here can beused to identify, map (two-dimensionally or three-dimensionally),observe, store, and use (for a wide variety of purposes) accurate andprecise information about the electrical signals occurring at specificidentifiable signal sources (for example, specific efferent nervesassociated with the contraction of the thumb and the extension of thefour fingers) at a given time or over a sequence of times. Theinformation about the electrical signals can be associated with specificindividual muscles, tendons, and nerves. The relative sizes ofelectrical potentials detected at different locations can be used toinfer which muscle or nerve is the source of an action potential. Asexplained later, when the electrical potentials are detected at multiplelocations on the skin, machine learning is one way to determine thespecific muscle or nerve that is electrically active. As also discussedlater, in some cases the determination can be made based simply onthresholds of amplitude. The relative amplitudes of the detecteddifferential signals at different locations can be used to infer whichmuscle or nerve is electrically active.

In examples in which electrical signals are determined to have occurredat specific nerves, the technology can effectively determine whichmuscle is the target of intended motion or is about to move without theneed to detect the electrical signals directly associated withcontraction of the muscles.

In some implementations, detected electrical potentials are notassociated with particular nerves controlling particular target muscles.In these other implementations, the nerve electrical signals aredetected at certain locations and are matched with known physiology tomap the detected electrical signals to corresponding target muscles.Electrodes help us map to particular muscles based on physiology. Inexperiments with people with ALS, who have atrophied non-moving muscles,the technology can detect nerve electrical signals and determine thatthey correspond to intended gross muscle movement even though there isno physical movement or muscle electrical signal present.

Therefore, of particular interest among the signal sources in, say, thewrist, and in the information that can be generated from the detectedelectrical potentials, are details about when and to what degreeelectrical signals occur at the specific efferent nerves that controlthe muscles of the hand at a given time or a sequence of times. Themuscles of the hand have an important role in how the human bodyinteracts (for example through the fingers) with the external world. Theinformation that can be generated by the technologies that we describehere can enable a wide range of devices, systems, processes, and otherapplications that rely on understanding when, how, and the extent towhich the muscles of the hand have been or are going to be contracted orrelaxed and on gestures and intent represented by those contractions orrelaxations.

The muscles of the body produce electrical signals when they contract orrelax. Those muscle electrical signals can be detected by surfaceelectrodes on the skin of the body in the vicinity of the muscle usingsurface electromyography (“sEMG”) and can be used for various purposes.A pair of skin electrodes (located at the upper forearm, for example) istypically used to detect a voltage differential as an indicator of theelectrical activity in the muscle. The detected voltage differential canbe used to determine that the brain intended to cause (or is about tocause or has actually caused) the corresponding muscle to, for example,contract. The contraction could be, for example, of a muscle of theindex finger that would have the effect of clicking a computer mousebutton if the hand were holding a mouse. And, even if the hand were notholding a mouse, a processor could translate the voltage differentialsignals from the muscles into the intended action (clicking the mouse)by causing a program to do what would have been done had a real mousebutton actually been clicked.

Although some existing systems can enable some such applications bydetecting electrical signals that occur in the contraction or relaxationof the muscles themselves, there are advantages in being able to detectdirectly and process information about the electrical signals in thenerves such as the efferent nerves that trigger the electrical signalsin the muscles. The electrical signals that occur at the efferent nervesare more direct, precise, earlier-appearing indicators of activity atthe brain than are the corresponding electrical signals generated bycontraction of the muscles.

Many applications are designed to cause actions (for instance, theselection of a menu item on a user interface) based on information thatis representative of intent of a user expressed by the signalspropagated by the brain to the efferent nerves. Existing electricalpotential detection systems use muscle contraction electrical signals asa proxy for inferring the intention signals propagated by the brain.Among other things, the nerve electrical signals detected and processedby the technologies described here are received slightly sooner than theresulting muscle contraction electrical signals, can be more precise asindicators of intention than the muscle contraction signals, and canavoid errors that can happen when a muscle contraction does not in factaccurately represent the intention carried in the nerve electricalsignal. The latter can happen for a variety of reasons, such as, whenthe muscles are not working or not working properly or have atrophied orare missing (for example, patients affected by ALS, stroke, oramputation).

The technologies that we describe here provide more accurate and precisemeasurements of nerve electrical signals and electrical signals of othertissues than any known system. As a result, more effective and newapplications for information about tissue electrical signals are madepossible. Among other things, characterizations of gestures or intentassociated with the tissue electrical signals can be more accurate dueto less cross talk among electrical signals generated by differenttissues, for example, when muscle contractions (and the associatedmuscle electrical signals) could be appearing in multiple muscles at thesame time.

As mentioned above, nerve electrical signals can be more accurate andfine-grained indicators of muscle contraction and relaxation than muscleelectrical signals. Yet nerve electrical signals are often of smallermagnitude than muscle contraction signals. And all electrical potentialsdetected on the surface of the skin are subject to “noise”, including 60Hz noise from nearby electrical devices, electrical signals from muscle,bones, and other tissues, variation in the electrical properties of theskin, and other sources of noise.

The technologies that we describe here use various strategies andcombinations of them to improve the signal-to-noise ratio for theelectrical potentials detected on the skin. In implementations, thesestrategies can include one or more of the following:

1. Electrical signals occurring in different types of tissue andoccurring in the body for different purposes may have different spectralcharacteristics. For example, the typical frequency range of musclecontraction electrical signals is 20 Hz to 300 Hz with dominant energyin the 50 Hz to 150 Hz range while the typical frequency range of nerveelectrical signals is 200 Hz to 1000 Hz. By applying appropriate filtersduring signal processing of the detected electrical potentials, it ispossible to enhance, for example, the relative strength of the nerveelectrical signals.

2. Some parts of the body (such as the wrist) contain relatively littlemuscle tissue and, in particular, relatively little of the bulky part ofmuscle tissue that produces the predominant portion of the forceresulting from contraction of the muscle. Placement of the electricalpotential detectors on the skin at parts of the body that haverelatively little such muscle tissue can reduce the relative amplitudeof “noise” that the muscle contraction electrical signals produce at theskin surface.

3. In some parts of the body (such as the wrist), the nerves for whichthe nerve electrical signals are to be detected lie between the detectoron the surface of the skin and muscle or other tissue. Therefore,electrical signals occurring at the other tissues may be blocked orattenuated by intervening tissue on their way to the skin. As a result,placement of the electrical potential detectors on the skin at such abody part can reduce the “noise” in the electrical potential signalsassociated with the nerve.

4. The way the electrical signals from multiple signal sources combineat the surface of the skin can be sensitive to the specific location orpattern of locations on the skin at which electrical potential detectorsare placed. There is a scientific basis for detector placement based onthe physiology and arrangement of nerves. In one location, the portionof the measured electrical potential attributable to nerve electricalsignals can have a lower signal-to-noise ratio than at a nearby locationthat is within a centimeter or less of the first location. Therefore,adjusting the locations of the detectors in small increments to locatethe highest signal-to-noise ratio locations can be useful.

5. In applications of the technologies that we describe here, movementsof parts of the body caused by contractions or relaxations of musclesare inferred from electrical potentials (and patterns of them) measuredat a given time or at a sequence of times at two or more locations onthe skin. Information about the inferred movements of parts of the bodycan then be provided to applications for a wide variety of purposes.Many different approaches can be used for making such inferences. Basicmachine learning algorithms like Fine Gaussian support vector machines(SVMs) when trained on a representative feature set are able to achievehigh accuracy while maintaining low computational complexity. As moreuser data is aggregated, the accuracy of the model will increase if thefeature set is representative of the actions being distinguished fromone another. If the extracted feature set is not representative or doesnot increase in accuracy more computationally intensive learningframeworks can be used to identify features. Deep neural nets have beenshown to be effective in identifying a concise and representativefeature set. However, this feature set may not rely on the underlyingbiology and may reduce the ability to make inferences about the system.Among such approaches are those in which known relationships between theelectrical potentials at the skin surface and corresponding actualmovements of parts of the body are used to train machine learningprocesses to recognize relationships between features extracted from theelectrical potentials and the corresponding movements of parts of thebody. Such known relationships can be obtained from people in variousways. The choice of which known relationships are used for training andthe choice of machine learning process to which they are applied mayaffect the speed and accuracy of the inferences. In someimplementations, useful relationships for training can be derived frompeople whose relevant muscles have atrophied or do not function for someother reason. In such cases, because muscle contraction electricalsignals do not occur, it is possible to reduce the impact that wouldotherwise be caused by such relatively large magnitude unwantedelectrical signals. In general, the nerve electrical signals thatcorrespond to putative muscle contraction in such people are the same asthe nerve electrical signals that correspond to actual musclecontraction in people whose muscles function properly. Historicaltraining can take advantage of this same principle.

6. Because muscle contraction electrical signals may represent asignificant source of “noise” and electrical potentials associated withthose electrical signals may be available at the surface of the skin, itis possible to disambiguate muscle electrical signals from aggregatedetected electrical potentials. The noise can be reduced by detectingthe electrical potentials at locations that are nearer to the musclesthan the locations where the nerve electrical signals are beingdetected. Then, the muscle electrical signals can be canceled from thenerve electrical signals.

7. Electrical signals from the brain (impulses, action potentials)propagate along nerves at speeds that are characteristically differentfrom the localized occurrence of electrical signals in muscle (whichcontract “in place”). Nerve electrical signals directed at target musclecontraction placement can travel as fast as 119 m/sec. Pain electricalsignals can travel more slowly, for example, at 0.61 m/sec, and touchelectrical signals can travel at, for example, 76.2 m/sec. By placingelectrical potential detectors (e.g. skin surface electrodes) of a givenpair in positions and relative orientations such that the electricalpotential change caused by a propagating impulse will appear first atone of the detectors and at a measurable time later at the other of thedetectors, it is possible to determine the direction of propagation andthe speed of propagation of the impulse. This information enables thetechnologies that we describe here to distinguish between muscleelectrical signals and nerve electrical signals. Similar approaches maybe useful to distinguish electrical signals of bones or other tissues.

8. Nerves include bundles of neurons some of which have motor functions(to cause muscles to contract or relax) and some of which have sensoryfunctions (to report sensed stimuli back to the brain). In detectingnerve electrical signals, it is useful to distinguish between motor andsensory nerve electrical signals. By detecting the polarity of adifferential signal between two electrodes placed nearer to and fartherfrom the fingers, it is possible to identify the direction ofpropagation and therefore to disregard sensory nerve electrical signalsbased on the polarity.

In order to reduce noise and improve the signal-to-noise ratio varioustechniques or components can be utilized. One such technique is to addconductive or magnetic material around the electronic components to actas a shield from electromagnetic interference. Capacitors can be addedinto the circuit to act as reservoirs to smooth voltage variations.Chokes, or inductances, can be added to the circuit to smooth currentvariations. Low pass filters, high pass filters, band pass filters, andband stop filters can all be created physically or digitally toattenuate noise in a signal. High pass filters only allow high frequencysignals to pass through them. Low pass filters only allow lowfrequencies to pass through them. Band pass filters only allow a certainrange from one frequency to another to pass through them. Band stopfilters attenuate frequencies in a range of one frequency to another andallow all others to pass through. The frequencies that these filtersallow or attenuate can be adjusted by altering the circuitry or softwareto the needs of the system. By combining noise reduction techniques withsignal enhancement techniques like those of an amplifier, whichincreases the amplitude of the signal, one can produce a morecomprehensive signal for analysis.

In general, in addition to detecting, measuring, sensing, and derivinginformation specifically about electrical signals occurring at nerves,some implementations and aspects of the technologies that we describehere can instead or also use electrical signals occurring in muscle orother tissues. Also, information provided by other sensors or othersources can also be used in combination with nerve electrical signals,including information from inertial measurement units (IMUs),gyroscopes, moisture sensors, temperature sensors, visible lightsensors, infrared sensors, cameras, audio sensors, proximity sensors,ultrasound sensors, haptics, skin impedance pressure sensors,electromagnetic interference, touch capacitance, external devices orother detectors.

HMI Platform

As shown in FIG. 9 , some implementations of the technologies that wedescribe here provide a comprehensive, integrated, universal humanmachine interface (HMI) platform 10 that, on one hand, can detect orotherwise receive and send signals, commands, or other information 12 toand from user interface devices 14 (such as the wrist device) associatedwith users 16, and, on the other hand, can send and receive signals,commands, or other information 18 to and from application devices andprocesses 20. Among other things, the HMI platform 10 can accumulate,analyze, alter, and otherwise process the information 12 and theinformation 18 in a wide variety of ways in order to create a usefulcomprehensive way for the users 16 to interact through the userinterface devices 14 with (we use the term “interact with” in itsbroadest sense) the application devices and processes 20.

The user interface devices 14 can comprise a wide variety of componentsincluding any kind of hardware or software 22 that can receive, process,and send signals or commands or other information from any part of thehuman body, such as from the fingers, hands, feet, legs, mouth, or eyes,or provide signals or commands or sensory stimulation or otherinformation to any of the sensory organs of the body, including theeyes, ears, nose, mouth, and skin. In some cases, the receivedinformation will have been provided as a result of motion of one ofthese parts of the human body, for example, by motion of a finger ormovement of an eye. Typically, the information provided to the user willbe provided in a visual, auditory, or haptic form, although other modesare also possible. In some cases, the user interface devices are inphysical contact with one or more parts of the user. In some cases, theuser interface devices (such as cameras or microphones) are not inphysical contact with one or more parts of the user but are able toperceive one or more parts of the user at an appropriate distance.

Among the broad range of kinds of user interface devices 14 that can beserved by the HMI platform 10 are keyboards, mice, track balls,joysticks, triggers, switches, touchscreens, dials, knobs, microphones,image sensors, cameras, glasses, headsets, gloves, items of clothing,speakers, earphones, displays, projectors, capacitance sensors, touchsensors, fingerprint detectors, moisture detectors, temperaturedetectors, motion tracking sensors, ultrasound sensors, phones,smartwatches, laptops, tablets, computers, televisions, vehicles,robots, exoskeletons, prosthetics, drones, health monitoring devices,health monitoring suits, electro-diagnostic systems, alarm systems,and—of particular interest to the discussion of certain examplesbelow—electrodes or other skin electrical potential detectors.

The demarcation between the HMI platform and the user interface devicesand between the HMI platform and the applications need not be rigid orstatic. In some implementations, one or more of the user interfacedevices (or parts of them) can conceptually be considered part of theHMI platform if the user interface device is specially designed to workeffectively with the HMI platform or can be considered conceptuallyexternal to the HMI platform if it was not specially designed to workwith the HMI platform. In some cases, a user interface device can haveelements that are internal to (part of) the HMI platform and otherelements that are external to (not part of) the HMI platform. Forexample, a wristband that detects raw electrical potential signals onthe surface of the skin could be conceptually external to the HMIplatform, while a signal processor for the electrical potential signalscould be conceptually part of the HMI platform even though located onthe wrist device.

The signals or commands or other information provided or received by agiven user interface device could have resulted from various differentlevels of processing and combinations of them. For example, the signalsor commands or other information could be raw electrical potentials,processed electrical potentials, or higher-level informationrepresentative of or derived from the processed electrical potentials orcombinations of them. The processing may have occurred entirely in theHMI platform, entirely in the user interface device, entirely in theapplication devices and processes, or could be split among them in avariety of ways.

Typically, when a user takes an action with respect to one or more ofthe user interface devices (we use the phrase “takes an action” in itsbroadest sense), the user has an intent (for example, an intention toclick the left mouse button). The intent is developed in the brain andexpressed in a gesture (for example, contracting the right indexfinger—the gesture—to cause the mouse button to be clicked—theintention) and the brain sends signals to nerves that propagate them tothe hand to perform the muscle actions (we use the phrase “muscleactions” broadly) that correspond to the gesture.

We use the term “gesture” broadly to include, for example, any muscleaction or (frequently) a suite of muscle actions that correspond (aloneor with one or more other gestures at a given time or a sequence oftimes) to one or more intentions of a subject. “Thumbs up”, “high five”,a “hand shake”, clicking a mouse button, flexing an upper arm, andobject dragging on a user interface are a few examples of gestures.Gestures can be well known and accepted as part of typical normal humanactivity or gestures can be newly developed for a variety of purposes. Ahand shake is a centuries old gesture, while a mouse click was createdas a gesture relatively recently.

An important activity of components of the HMI platform is to identify(infer, determine) gestures 24 (using a gesture identification component26) and provide information about the gestures through a communicationcomponent 28 to the application devices and processes 20 where they canbe used in a wide variety of ways. In some cases, the applicationdevices and processes 20 use the gesture information to infer the user'sintent and then act on that intent. In some cases, an intentionidentification component 30 of the HMI platform can infer (determine,identify) a user's intent 32 for example, from one or a combination orsequence of gestures occurring at a moment in time or in a sequence overtime, and provide the intention information through the communicationcomponent 28 to the application devices and processes 20.

In some cases, the gesture or intent determined by the HMI platform neednot be “pushed” to a particular application. The HMI platform cantemporarily store the gesture or intent information and provide an APIor other mode of making available to any other process, program, orapplication that may request the gesture or intent information. In thatsense, the HMI platform can operate as a service provider, in some casesthrough an operating system. For example, if the HMI platform isimplemented as an app on a smart phone, the app can make available thegesture or intent information upon request to any other authorized apprunning on the smart phone.

In some examples, the flow of information from the user interfacedevices to the application devices and processes begins with the brain52 of a user formulating an intention 54 to take an action with respectto one of the user interface devices 14 or with respect to one of theapplication devices and processes 20 or both. The brain 52 converts theintention into a gesture (or two or more gestures at a given time orsequence of times) 24 that often involves moving one or more muscles 56in a way to implement the gesture and in turn express the intention. Insome cases, the brain causes appropriate nerves 58 to propagate signals57 to appropriate muscles to cause them to move in ways that correspondto the gesture. One or more aspects of the gesture can be detected bythe user interface devices. For example, the user interface device canbe a wrist device having electrodes to detect electrical potentials onthe surface of the user's skin that correspond to the electrical signalspropagated along the nerves. In some instances, the user interfacedevices detect the muscle motions in other ways, such as by camera orinertial measurement units. The electrical signals or other events for agesture as detected by the user interface devices can occur at a momentin time or a sequence of such moments.

The electrical potentials appearing on the skin, or other indications orinformation about muscle motion, are acquired, sensed, or otherwisedetected at the user interface device by one or more detectors 64.Typically, the raw electrical potentials or other indications of musclemotion 65 must be processed (e.g., in a signal processor 66) to, forexample, remove noise, remove signals that are not useful, organize andorder the signals, and express resulting processed signals in formatsthat will be useful for the next steps. Other major functions of thesignal processor include but are not limited to, measuring, filtering,or compressing, analog signals that are generated as the output of theuser (in signal form) for the input of the device.

We use the term “signal processing” broadly to include, for example, anyoperations performed on relatively raw signals to produce relativelymore processed signals that are more accurate, concise, precise,compact, complete, relevant, or understandable, among other things.

The resulting processed signals 68 can then be stored in a storage 70and (whether or not first stored) sent to the HMI platform by acommunication component 72. In the HMI platform, the processed signalscan be used by the gesture identification component or the intentidentification component to identify corresponding gestures or intent orboth of the user. In some cases, the gesture identification componentand the intent identification component can be combined as a singlecomponent. In some instances, one or the other of them need not beincluded. In some implementations, the two components can operate inparallel.

The gesture identification component and the intent identificationcomponent can receive and use combinations of processed signals from twoor more different user interface devices or detectors and can apply oneor more algorithms, analytics, transformations, or other interpretationprocesses to interpret, identify, or determine the gestures and intentof one or more users.

A wide variety of gesture or intent identification processesrepresenting a range of different complexities and styles can be used.In some cases, the identification process can be as simple asdetermining that an electrical switch in a computer mouse has madecontact and identifying that the user's gesture was to click the mousebutton. In some instances, the identification process can be a morecomplicated method needed to interpret, for example, a set of nerveelectrical signals detected on the surface of the skin of the wrist ofthe user. Classifiers or other machine learning components can be usedin such cases. Basic machine learning algorithms like Fine Gaussiansupport vector machines (SVMs) when trained on a representative featureset are able to achieve high accuracy while maintaining lowcomputational complexity. As more user data is aggregated, the accuracyof the model will increase if the feature set is representative of theactions being distinguished from one another. If the extracted featureset is not representative or does not increase in accuracy morecomputationally intensive learning frameworks can be used to identifyfeatures. Deep neural nets have been shown to be effective inidentifying a concise and representative feature set. However, thisfeature set may not rely on the underlying biology and may reduce theability to make inferences about the system. By training such machinelearning processes based on known relationships between nerve electricalsignals and corresponding gestures or intent, it is possible to applythe trained process at “run time” to a new set of nerve electricalsystems from a user to derive a correct interpretation of acorresponding gesture or intent of the user. As mentioned earlier, insome cases, features of a feature set are extracted from known data andcorrelated with known gestures as the basis for training and thenapplying the machine learning techniques.

As mentioned earlier, in some implementations, the HMI platform includesone or more intent identification components that can use the identifiedgestures or other processed signals or both to infer the intent of theuser that corresponds to one or more gestures. Machine learningtechniques can be used for this purpose. For example, a gestureidentification component may determine that processed electrical signalscorrespond to a gesture in which the user is waving his hand in the airwith his index figure extended, and the intent identification componentcould use that interpreted gesture to infer the intent of the user to bepicked up by a taxi.

The HMI platform can serve a single user interface device of a singleuser (e.g., a wrist device) and a single application device or processof the user (e.g., augmented reality glasses), or it can serve multipleuser interface devices, multiple users, multiple application devices orprocesses, or combinations of them. For example, a user may have a wristdevice and a track ball as user interface devices and a smart phone anda laptop computer as applications.

Communications between the HMI platform and each of the user interfacedevices and each of the application devices and processes can conform toagreed protocols and formats for the signals or other information beingcommunicated. In some implementations, the HMI platform iscustom-configured to communicate using protocols and formats that arepeculiar to a user interface device or an application device or processbeing served. In some instances, the HMI platform can communicate usingprotocols and formats that are device-independent andprocess-independent and can serve essentially any user interface deviceor any application device or process that conforms to the protocol orformat independently of its design or manufacture. For this purpose,components of the HMI platform can expose one or more applicationprogramming interfaces (APIs) or both to user interface devices orapplication devices and processes. In some cases, the APIs can definestandardized formats and data structures to be used to express signalsor commands or other information that are passed back and forth betweenthe HMI platform and the user interface devices or the applicationdevices and processes. (For simplicity, we sometimes will use the word“application” to refer broadly to any such device or process.)

Wrist Device

As shown in FIG. 10 , some examples of user interface devices are in theform of a wrist device 802. The wrist device can include, for example, agenerally rigid component 804 connected to a flexible, adjustable wristband 806 configured to permit the inner circumference 809 of the wristdevice to be adjusted to different size wrists much as a watch band canbe adjusted to the wrist of its user.

In some implementations, the rigid component 804 includes a housing 806on which are mounted a circuit board 808 holding electronic components,a battery 810, a user interface panel 812, a communication (e.g., USB)port 814, an external electrode port 816, (e.g., quick release) wristband mounting pins 818, 820 to lie within channels of the wrist band,and a set of electrodes 826 held in an electrode cartridge 822.

The housing 806 is generally configured to rest on the top of the wristwith the wrist band wrapped around the bottom of the wrist. The housingis designed to provide a rigid inner surface 828 having two segments830, 832 that cooperate to keep the wrist device registered in apredefined position and orientation with respect to the relatively flattop surface 834 of the wrist and to the outer side 836 of the wristalong the side of the ulna, and also registered against the styloidprocess 838 of the ulna. As a result, although the wrist device can beworn on many different sizes and shapes of wrists, it will reliably bein a known position and orientation to detect electric potentials at theupper surface of the wrist produced by electrical signals in the ulnarnerve and the median nerve. This is because the ulnar and median nervestend to be positioned at roughly the same distance from the side of theulna for wrists of many different sizes and shapes. Consequently, theelectrodes (described later) that are exposed at fixed positions at therigid inner surface will reliably detect electric potentials at the skinthat are generated by known nerves in the wrist.

To achieve this automatic registration on wrists of various sizes andshapes, the segment 830 has a flat rigid inner surface 840 and thesegment 832 has a concave curved rigid inner surface 842 that meets theflat rigid inner surface. The contours of inner surfaces 840 and 842tend to urge the wrist device into a consistent position and orientationrelative to predetermined locations along the nerves that control thehand muscles for wrists that have a variety of sizes and shapes.

The electrodes 826 are exposed at the rigid inner surfaces 840, 842 sothat contact surfaces 844 of the electrodes face the skin of the wrist.A wide variety of sizes and shapes of contact surfaces 844 are possible.In some cases, the contact surfaces are round and can have widths in arange as small as one millimeter (or smaller, in some examples) or aslarge as ten millimeters (or larger, in some instances). Typically, thecontact surfaces will be flat but other contours may be useful to matchcontours of the skin, for example. Different ones of the electrodes canhave different sizes and contours.

In some implementations, the contact surfaces are arranged in a pattern.The pattern can comprise pairs 846. In some cases, each pair can bepositioned along an axis 848 that is parallel to the length of theforearm with one electrode of the pair closer to the fingers than theother electrode of the pair. The number of contact surfaces can be asfew as two and as many as tens or hundreds. In particularimplementations, there can be between two and eight pairs of the contactsurfaces arranged in a series along the rigid inner surface. The spacingbetween the two contact surfaces of each pair can vary; in some cases,the spacing could be in a range as small as five millimeters or as largeas 25 millimeters or more. The spacings between adjacent pairs ofelectrodes can vary.

A wide variety of other patterns of electrodes and their contactsurfaces can be used including patterns of regular rows and columns andpatterns that are irregular along either or both of two dimensions.

In some implementations, some of the electrodes can have contactsurfaces exposed on the inner surface of the flexible wristband 806 forcontact with other parts of the skin of the wrist, including the bottomof the wrist. Such electrodes can be organized in pairs in regular rowsand columns or in other patterns and can number as few as one electrodeor as many as tens or hundreds. The contact surfaces can have varioussizes, shapes, dimensions, and contours as indicated for the contactsurfaces described above.

In general, the numbers, sizes, shapes, contours, and patterns oforganization of the electrodes can be chosen empirically, for example,to provide the most accurate, concise, precise, strongest, and leastnoise-influenced electrical potential signals from the skin of the wristand to provide the most accurate inferences from such potential signalsof contractions or relaxations of particular muscles, gestures, andintent.

By using a very large number of closely spaced electrodes to captureelectrical potentials from a closely spaced array of locations on thesurface of the skin at a given body part over time, it can be possibleto derive accurate information about the action potentials of (every)nerve, and muscle in the vicinity of the measuring electrodes over time.That information can be useful for a variety of purposes.

For purposes of carrying electrical current from the contact surface ofeach of the electrodes to the signal processor 66 wires, conductiveruns, or other types of electrical couplings are provided between theelectrodes and circuit board 808. Typically, each of the electrodes willhave its own such electrical connection so that the electrical potentialon that electrode can be detected and measured independently of theelectrical potential on any of the other electrodes. In some cases, twoor more of the electrodes can be connected through a common electricalconnection to the circuit board. As shown in FIGS. 12, 13, and 14 , eachof the electrodes can include a snap nipple 841 on the opposite face ofthe electrode from the contact surface 844. The snap nipple mates with asnap socket 843. The snap socket is mounted loosely in a sleeve 845 ofthe electrode cartridge to be slidable along the axis of the socket. Ahelical spring 847 is soldered to the snap socket.

Any one or more of the electrodes can be mounted on or at the rigidinner surface of the housing or the inner surface of the flexible wristband. One or more of the electrodes can be mounted to be resilientlyurged toward the skin when the wrist device is worn. As shown in FIGS.12, 13 , and 14, in some cases, the resiliency of an electrode can beachieved by mounting it in the sleeve 845 in the cartridge 822 andincorporating the helical spring 847 to urge the contact surface of theelectrode in the direction of the skin. The direction in which theresiliently mounted electrode is typically urged is normal to the innersurface of the wrist device but could be in a non-normal direction. Thespan of the motion of each of the resilient electrodes can be in therange of from, say, one millimeter or less to five millimeters or more.

As a result of these features, when the wrist device is put onto thewrist each of the resiliently mounted electrodes will automatically forma good electrical contact with the skin and will accommodate smalldifferences in the distances between the inner surfaces of the wristdevice and the skin of the wrist at various locations.

Other configurations can be used for the wrist device and its componentsto achieve resiliency of the electrodes. For example, the housing or aportion of the housing (for example the portion that faces the skin) canbe formed of a flexible resilient material (such as rubber) and theelectrodes can be mounted directly in that material with a short sectionof the electrode projecting from the inner surface toward the skin. Whenthe wrist device is put onto the wrist, the flexible resilient materialurges the contact surfaces of the electrodes toward the skin tocontinuously provide good contact with the skin when motion of thewrist, for example, causes small changes in the small gap between thesurface of the skin and the inner surface of the housing.

As noted above, in some cases, the electrodes can be attached to anddetached from the wrist device using a snap to permit easy replacement.In some implementations, mounting of the electrodes can use disk magnetsembedded in the inner surface of the wrist device, a flexible circuitwith connection runs is embedded within the wrist device, and theelectrodes fit within divots provided in the inner surface of the wristdevice and have magnets to couple to the disk magnets.

The electrodes can be formed of a variety of metals, metal alloys, orconductive materials, but certain ones are more effective than others.Regarding metal and metal alloy, silvers, coppers, golds, aluminums,zincs, nickels, and brasses are several of the more conductive ones.Electrodes formed with these materials work especially well becausetheir electrons move more freely than that of other elements or alloys.This causes any electrical current exhibited on these materials totravel with little resistance. Some other conductive materials andcompounds include silver/silver chloride (Ag/AgCl), conductive fabricsand threads, and graphene. In some implementations, the electrodes areused in a dry contact mode in which the contact is made directly betweenthe metal of the electrode and the skin. Conventional dry contactelectrodes tend to produce high impedance coupling between theelectrodes and the skin, which makes it difficult to acquire goodelectrical potential signals. This issue is addressed in wet electrodetechnology by providing a liquid or gel conductive material thatproduces a low impedance coupling to the skin, but has the disadvantageof being messy, among others. Some implementations of dry electrodesthat we describe here use a conductive gel and a permeable membrane thatpermits exchange of ions between the skin and the conductive gel whileblocking moisture from wetting the outside of the membrane. Suchelectrodes are described in U.S. Pat. No. 8,798,710 (the contents ofwhich are incorporated here by reference) and are available fromCognionics of San Diego, Calif., USA.

In some examples, one or more of the electrodes can be mounted rigidlywithout resilience.

In some cases, the registration of the wrist device on wrists ofdifferent sizes and shapes can be achieved by techniques other thanrigid features. For example, housing structures with flexible joints,hinges, or modular components can accommodate different wrist sizes andshapes. The device could have elements for resizing like an adjustablewrench, like a clamp, or like a clam-shell. Segments or lengths of thehousing could be added or removed to extend or reduce the overall sizeof the housing as in linkages of a watch.

Through the external electrode port 816, a wire 817 can be connectedfrom the housing 206 to an external “stick-on” electrode 856 that can beattached to the skin of a subject by tape or other adhesive techniques.The external electrode 856 can be of the kind that uses the permeablemembrane. In some cases, the external electrode can be applied using aconventional conductive liquid or gel.

The port 816 could be arranged to provide independent connections to twoor more such external electrodes.

In some cases, the external electrode or electrodes can be used insimilar ways and for similar purposes to the electrodes mounted on thewrist device. In some cases, the external electrode or electrodes can beused in different ways and for different purposes. For example, theexternal electrode could be placed in a position or orientation todetect muscle electrical signals that could be used to offset or cancelmuscle electrical signal noise in one or more other electrodes. In someinstances, the external electrode or electrode can be used forexperimental or study purposes.

The USB port 814 can receive a USB connector 815 for purposes ofcommunicating data, information, and commands between the wrist deviceand, for example, the HMI platform, and to carry current to charge thebattery in the wrist device.

The circuit board 808 can take a variety of forms, shapes, or sizes thatare consistent with the mounting of the wrist band on a wrist.Electronic components can be mounted on the circuit board.

As shown in FIG. 16 , in some implementations the electronic componentson the circuit board of the wrist device include analog components 860,digital components 862, and one or more analog-to-digital converters 864that convert analog signals that are captured by the electrodesprocessed by the analog components for delivery to the digitalcomponents (e.g., one or more digital processors 866) for furtherdigital processing.

The processor 866 uses digital signal values 868 received from the oneor more analog-to-digital converters. The digital signal values arebased on electrical potentials at the respective electrodes at momentsin time and sequences of moments in time.

In some implementations, the digital signal values representdifferential potentials, that is, the difference between the electricalpotential at one electrode 873 and the electrical potential at a secondelectrode 875, both relative to a ground potential at a third electrode877.

For this purpose, the two electrodes of each differential pair can beconnected to and served by a dedicated signal path 870, 872 from theelectrodes to a shared analog-to-digital converter 864.

Each of the digital signal paths includes a notch filter 874, 876 thatremoves, for example, 60 Hz frequency components of the electricalpotential signals detected at the skin by the two electrodes of thedifferential pair, to reduce or eliminate the noise caused by 60 Hzpower sources in the vicinity of the subject. The notch-filtered version878 of each electrical potential signal is delivered to aninstrumentation amplifier 879 the output of which is the value of thedifference of the electrical potentials at the two electrodes of thedifferential pair. In determining the difference of the electricalpotentials, each of the electrical potentials is considered relative toa common ground potential represented by the electrical potential at athird particular electrode that is typically selected to be away fromthe locations on the wrist where good electrical potentials from nerveelectrical signals can be detected. Therefore, the difference of theelectric potentials is the difference of two values, one being thedifference between the electrical potential at one of the two signalelectrodes and the electrical potential at the ground electrode, and theother being the difference between the potential at the other of the twosignal electrodes and the electrical potential at the same groundelectrode. In some implementations, the ground electrode is pre-selectedand cannot be changed. In some cases, any of the electrodes can beselected as the ground electrode 877.

Similarly, in some cases, the processor can select which electrodescomprise each of the differential pairs of electrodes and which of thepairs are being actively used (polled) by the processor at a given time.In the case of multiple processing channels, one channel could receiveelectrical potentials from a specific three of six available electrodes;the other channel could be switchable between the stick-on electrode(connected to the audio jack) or one of the electrodes having a contactsurface exposed from the inner surface of the wrist device. By utilizingmultiple channels there is more access to raw signal information as thedata is collected from each corresponding set of electrodes. This datacan be processed and utilized for discerning more complex gestures,collecting data for research studies, comprehensive disambiguation ofmuscular or nerve signals as well as other applications.

The gain of the instrumentation amplifier is governed by a programmableresistor 880, the gain of which is controlled by the processor. Thisdynamic gain of the instrumentation amplifier provides an impedancematching function to match the impedance between the two electrodes ofthe differential pair. The instrumentation amplifier is also connectedto ground by virtue of the third electrode 877.

As shown in FIG. 19 , dynamic impedance matching of the instrumentationamplifier 980 to the changing impedance of the electrode-skin interface982 helps to improve the signal-to-noise ratio of the signals acquiredfrom the electrode. A dynamic impedance matching component 984 receivesmeasurements 986 of the impedance of the electro-skin interface. Theimpedance measurements are done by a measuring component 986. Based onthe impedance measurements, the dynamic impedance matching componentcalculates an ideal resistance (impedance) of the instrumentationamplifier input needed to improve (e.g., maximize) the signal-to-noiseratio. An output 988 of the dynamic impedance matching component 984then controls the resistance of the programmable resistor 880accordingly.

The output of the instrumentation amplifier, the differential potentialvalue 882, is provided to a band pass filter 884, which passes a band offrequency components of the differential potential value that arecharacteristic of electrical signals appearing at a kind of tissue ofinterest. In some examples, the tissue of interest is nerve tissue andthe band pass filter can pass signal frequency components in the band 20Hz to 1000 Hz, which removes some frequency components characteristic oftissues that are not of interest. For example, muscle electrical signalstypically have frequencies in the range of 5-400 Hz, nerves havefrequencies in the range of 200-1000 Hz, including brain electricalsignals (the frequency of individual neurons firing, rather than EEGbrain waves (alpha, beta, gamma) which are the lower frequency (0 Hz to80 Hz) oscillation patterns of aggregated nerves.

The band-filtered version of the differential signal 886 is provided toa gain amplifier 888 in its original form 890 and also to the gainamplifier in a rectified form 892 after having been passed through arectifier 895. The gain amplifier uses as its input the original signalif it has a positive polarity and the rectified form if the originalsignal has a negative polarity. The gain of the gain amplifier iscontrolled by the value of a programmable resistor 894, the value ofwhich is determined by the processor. The processor determines this gainlevel based on the output signal voltage. Outputting the signal this wayprovides data at a measurable voltage level for easier visualization andanalysis.

When a propagated electrical signal (action potential) appears at alocation along a nerve, electrodes that are closer to that location willdetect a higher electrical potential on the skin than electrodes thatare farther away. When a propagated electrical signal (action potential)appears at a location along a nerve, a polarization phase occurs and theelectrical potential changes when reaching a threshold. The electricalpotential moves towards a positive orientation. Then, as the actionpotential at that location goes from an excited state to a restingstate, during a so-called repolarization phase, the electrical potentialwill go to zero and swing briefly to a refractory period, or a morenegative potential than its initial state, before converging at aresting state. The electronic components take advantage of this effectto determine the direction of propagation of the action potential inorder to disambiguate motor action potentials from sensory actionpotentials, and to determine the speed of propagation.

The analog-to-digital converter is continually receiving processedversions of the differential electrical potentials detected at each ofthe differential pairs of electrodes on the skin. The analog-to-digitalconverter is controlled to sample only one of the differentialelectrical potentials from one of the pairs of electrodes at a giventime and to sample the different pairs of electrodes in, for example, around-robin mode. The analog-to-digital converter can be arranged tosample input signals at any rate that is high enough to capturefrequency domain information of interest carried by each of the inputsignals and to capture that information for as many differential pairsof electrodes that are served by the analog-to-digital converter. In twospecific examples, the rate of polling a differential pair of electrodescould be 500 Hz or 4000 Hz or higher depending on the specifications andlimitations of the analog-digital converter. One analog-digitalconverter can serve one differential pair of electrodes or many, inwhich case the sampling rate of the analog-digital converter would beshared among the pairs. The polling rate should typically be at leasttwice the frequency of the signal being detected. In someimplementations, more than one analog-to-digital converter could be usedto serve corresponding sets of electrode pairs in parallel.

In typical implementations, therefore, the processor (we sometimes usethe singular “processor” to refer also to examples that use two or more“processors” in parallel, for instance) will receive streams of digitalpotential values representing differential electrical potentials at therespective differential pairs of electrodes at successive moments intime. In some cases, the processor is a special purpose processor thatserves as a characterization engine to characterize motions, gestures,or intent based on the digital potential values. We sometimes use theterm “processor” to refer to such special purpose processors orcharacterization engines.

Each digital potential value can be, for example, a 24-bit value (if theanalog-to-digital converter has a 24-bit resolution) tagged with anidentifier 902 to associate it with one of the differential pairs ofelectrodes and also tagged with a time stamp 904.

In addition to receiving and using the digital potential values providedby the analog-to-digital converter, the processor can receive and useother digital source values 905 received from a wide variety of otherinput sources 907, such as 9-axis or other inertial measurement units(IMUs), gyroscopes, moisture sensors, temperature sensors, lightsensors, cameras, audio sensors, location sensors, and proximitysensors, to name a few. The IMU can provide data about a user interfacedevice being tapped or jostled either by the user as an intentional orunintentional input which can be used as a digital source value. One ormore of these digital source values can be used by the processor inimplementing processes that generate the characterizations mentionedearlier.

The incoming digital potential values and the incoming digital sourcevalues (we sometimes use the term “digital values” to refer to both thedigital potential values and the digital source values) can be storedtemporarily in memory 906 mounted on the circuit board. The storage canbe of a size that is large enough to serve as a buffer but not to storelarge volumes of digital values for study or later use. In some cases,the storage can be large enough to temporarily store a substantialvolume of digital potential values during periods when the wrist devicecannot communicate with other parts of the HMI platform, for latercommunication to the HMI platform. In some instances, the storage can belarge enough to store an entire set of digital values to be used by theprocessor or by other parts of the HMI for a particular task. Portionsor all of the stored digital values can be transferred wirelessly by thewrist device to other parts of the HMI, such as to a local smart phoneas needed. Bluetooth or any other wireless protocol can be used.

The processor on the wrist device can process the digital valuesessentially in real time as they are detected, or by fetching them frommemory as needed, to produce characterizations 908 of musclecontractions or relaxations, gestures, or intent, or combinations of anyof them. We use the term “characterization” broadly to include, forexample, the digital potential values themselves, translations ormodifications of them, and any identifications, descriptions,inferences, or other indications or quantifications of musclecontractions or relaxations, gestures, or intent or combinations ofthem. After processing the digital potential values, the resultingcharacterizations can be stored in the storage on the wrist device, orcommunicated to another component or portion of the HMI, for example, toone or more applications or for any of a wide variety of other purposes.

As one specific example, a particular set of digital values can beprocessed by the processor to generate one or more characterizations ofa contraction of the index finger of a user corresponding to a motionused to click a button of a mouse. The characterization of the gesturecan be sent wirelessly essentially in real time to a local computerrunning an interactive application where the characterization can betreated as a mouse click event even though the user has only engaged ina mouse click gesture but has not actually clicked a physical mousebutton.

In some cases, the characterizations can represent quantifications ofspecific two-dimensional or three-dimensional positions or orientationsor changes in position or orientation of a part of the body of the user,for example, a tip of a finger. Because of the accuracy, precision, andquality of the digital potential values, the quantifications can be moreprecise than known other techniques and may be, for example, at aresolution of one millimeter or smaller. The accuracy of thequantification can be affected by the size of nerves, the size ofmuscles, and the size of electrodes. In some cases, the resolution maybe in the range of ten millimeters.

The processor can use a wide variety of algorithms, techniques,computations, translations, mappings, transformations, classifiers,statistics, models, analyses, or other processes and combinations ofthem to generate the characterizations from the digital values. Theprocesses used can depend on the processing power of the processor, thestorage capacity of the memory, the number, type, and rate of receipt ofthe input digital values, the complexity of the characterizations to begenerated, and other factors. The processes and portions of them can beimplemented in a variety of ways, including as software executed by aspecial purpose processor, as firmware, or as dedicated hardware. In thecase of software or firmware, the processes or portions of them can beinstalled at the time of manufacturer or can be installed or updatedafter manufacture, for example, by a wireless download or a wireddownload through the communication port (e.g., a USB port). Such updatescan improve or otherwise alter the performance and operation of theprocesses or portions of them based on knowledge developed over time(for example using historical data about the correlation of electricalpotentials and corresponding characterizations), can take account of newand additional data sources available to the processor, and can servenew or additional applications.

In a relatively simple example, the processor could perform thefollowing steps to generate a characterization of a gesture based on24-bit digital potential values from the analog-to-digital converter.The 2{circumflex over ( )}24 possible digital potential values can beclassified using three threshold states (LT [construed as the hand atrest], MT, and HT), and the classifications can be used to derive acharacterization of a gesture as follows. A sequence of a digitalpotential value going above LT and later falling below LT ischaracterized as gesture 1; a sequence of a digital potential valuegoing above MT and later falling below LT is characterized as gesture 2;and a sequence of a digital potential value going above HT and thenfalling below LT is characterized as gesture 3.

In some cases, two or more digital potential values can be used tocharacterize a gesture. For example: a sequence of digital potentialvalue 1 going above LT and digital potential value 2 going above LT andthen digital potential value 1 going down to LT is characterized asgesture 1; a sequence of digital potential value 1 going above LT anddigital potential value 2 going above MT and then digital potentialvalue 1 going down to LT is classified as gesture 2; a sequence of thedigital potential value 1 going above LT and digital potential value 2going above HT and then digital potential value 1 going down to LT isclassified as gesture 3; and a sequence of digital potential value 1going above MT and digital potential value 2 going above LT and thendigital potential value 1 going down to LT is characterized as gesture4. LT, MT, and HT are the values being compared to a low threshold,mid/medium threshold, or high threshold value in order to understand thecorrelation between a motion and the amplitude of the motion.)

If sufficient processing power is available, more complicated sets ofgestures (and intent) can be determined from more complicated sets ofdigital potential values and sequences.

In some processing-intensive examples, the processor executes aclassifier process 910. The classifier process and classifier data 912used by the classifier process are loaded either at the time ofmanufacture or later. In some cases, the classifier data identifies (a)sets of sources of digital values (such as a set of digital values fromparticular electrodes or specified IMUs), (b) specific digital values orranges of those digital values (the sets of sources and the specificvalues or ranges of values are sometimes referred to as “feature sets”)and (c) corresponding characterizations (such as muscle contractions orrelaxations, gestures, or intent) that are known to be correlated to thefeature sets. In effect, the classifier captures information about whichfeature sets correlate to which characterizations. At “run time” theclassifier process tests incoming sets of digital values and sequencesof them according to the classifier data, and maps them to correspondingoutput characterizations and sequences of them based on the classifierdata. The output characterizations and sequences of them can be storedor forwarded to applications or parts of the HMI for a wide varietypurposes.

Other types of characterization processes can be used. For example,frequency domain analysis can be used to characterize different actionsthat are performed, due to the different frequencies in thebiopotentials. Common average reference (CAR) can also be used tocharacterize different actions when multiple electrodes are in place.Simply by comparing one electrode with the average of all the electrodesaround the wrist to characterize activation base on the electrodelocation. Small and large Laplacians can be used to characterize theaction by comparing the sum of the nearest electrodes to characterizethe actions.

The wrist device has an outer surface that provides a user interfacepanel 812 for interaction by the user. As shown in FIG. 15 , in someexamples, the user interface panel includes two buttons 823 and 825. Thefunction of button 823 is to control the power of the device, either onor off. The function of button 825 is to provide Bluetooth pairingcontrol and to turn the Bluetooth controller on or off. An LED 827between the two buttons provides the following functions: indicatesstatus of the device including power on, Bluetooth pairing, as well asvisual feedback of sensor data including neuromuscular signal amplitude,IMU, haptics, and sensor fusion when combining data from multiplesensors.

In some examples, the user interface panel is a touch sensitive OLEDpanel that allows for visualization of data values and characterizationsand input features such as a keyboard. The panel can display the rangesof information that a smartwatch, phone, smart glasses, and computerwould. The panel can provide data such as charting, menu selections,mode changes, status of device operation, pairing status, battery life,videos, apps, internet sites, and content from tethered devices such assmart glasses or phones, and general computing. The user can interactwith the OLED in the same methods as those used by smartwatches, phones,smart glasses, and computers including tap, touch, swiping, or slidingone or more fingers or stylus. The OLED panel will enable the device tohave a low power display and have readability in sunlight with a higherquality illumination. Flexible display panels can be used to provideless rigid design options.

For purposes of enabling a developer to create software, hardware, andfirmware for use by the processor of the wrist device, a developer's kitcan be provided in connection with the wrist device. The developer kitenables the user to define what data values will be stored on the wristdevice, for how long, and in what format; when and how the data valueswill be delivered to other parts of the HMI, what processes will be usedto derive characterizations from the digital values, and how thecharacterizations will be expressed. The developer's kit can also enablethe developer to define analytics to be applied to the data values andto the characterizations by components of the HMI, to define elements ofa user interface to be presented to the user on the wrist device, and tospecifically define mappings between features sets and correspondingcharacterizations.

In some cases, the wrist device can contain or be combined with otherfunctions or features such as those provided by a watch, or a smartwatch, or an exercise monitor, or a health monitor that monitors orprovides feedback with respect to pulse rate, blood pressure,temperature, skin moisture, humidity, barometric pressure, acceleration,inertia, orientation, skin pressure, blood oxygenation levels, or otherparameters. In some cases, these functions can operate independently ofthe functions related to tissue electrical signals. In someimplementations, the functions can be combined or the data generated bysuch functions can be used as additional data sources for the processorin the user interface device.

App

As shown in FIG. 17 , in some cases, one or more features of the HMIplatform can be implemented in an app 926 running on a smart phone,tablet, laptop, or other processing device 928, for example. In somecases, these features can be implemented in a processor that is part ofthe wrist device. In some instances, the features can be implemented ina combination of the user interface device and the HMI platform.

The features 930 can be implemented in hardware, firmware, or software,or combinations of them. Each feature can be implemented by a module. Ingeneral, the features can involve user interaction through a userinterface 932 of the device 928 and can enable the user to customize,control, modify, configure, invoke, terminate, and otherwise use each ofthe features.

One of the feature modules can enable a user to create or customizegestures that the gesture identification component can identify. Forexample, the user can define a new gesture by instructing a gesturecustomization module 934 to accept a new gesture and then performing thegesture. As the gesture is performed, the data values from the processorin, for example, the wrist device are collected and stored andassociated with the gesture being defined. The user can then associatethe defined gesture with an action to be taken by other apps when theyreceive an indication that the gesture has been performed. For example,the new gesture could be the making of a fist, and the user could causethis gesture to be associated with the command to close the targetapplication. The user could be prompted to perform the gesture a numberof times so that additional sets of data values associated with thatgesture can be stored and, say, used to train a classifier. The gesturecustomization module also could be used to update or modify an existinggesture, for example, by enabling the user to update the action to betaken by the target app when the gesture is identified or to classifyslightly different physical motions as representing the gesture (saymaking of a hard first versus gently closing the hand).

One of the feature modules can be an outcome customization module 936that enables a user to identify the action to be taken by the targetapplication with respect to any existing or newly created gesture. Forexample, if the gesture is the closing of the index finger, the actionto be taken could be clicking a mouse button, typing a character, oranother action, as determined by the user through a user interfacepresented by the outcome customization module.

One of the feature modules can be a communication module 938 thattranslates gestures into corresponding communications so that a user cancommunicate through a variety of apps without using a keyboard or voiceas is typically done. The communication module can accept gestures fromthe gesture identification component and generate correspondingcommunications or elements of them for use by another app such as atexting app, a word processing app, a speech synthesis app, or a socialnetworking app, for example. The mapping of gestures to communicationscan be based on a standardized pre-defined set of relationships. In somecases, the user can view, update, or modify the mappings or create newmappings through a user interface presented by the communication module.

One of the feature modules can be a biopotential tracking and monitoringmodule that translates the electrical signals generated into an outputvisualization. This visualization would show, in some cases, thefrequency and amplitude data produced by the user in a graph-likemanner. This visualization would update over the period of use of thedevice. For example, a patient with a degenerative muscular disorderwould wear the device and be able to monitor their signal output overtime alongside their clinician. In some cases, the user can view, move,or update the graphs manually through a user interface.

One of the feature modules can be body composition analysis module. Fatand water have different impedances when exposed to an electricalcurrent. Lean tissue is mostly water and has a much lower impedance thanfatty tissue. This allows the measurement of fat content and watercontent in the body along with further health metrics. For example, if asmall amount of current is sent through one electrode in the wristmounted device to the user it can be received at another. It is thenpossible to calculate an individual's fat free mass and body fatpercentage based on the impedance values measured and that individual'sheight and weight.

One of the feature modules can be an activity tracking module thattranslates the electrical signals generated into an outputvisualization. This visualization would show, in some cases, thefrequency an individual has activated certain neuromuscular areas andhow those activations relate to resting or exercise values in agraph-like manner. This visualization would update over the period ofuse of the device.

One of the feature modules can be activity identification module inwhich the device recognizes the activity performed by the user. Forexample, the user closes their first to a certain point and flexes theforearm and bicep to do a pullup exercise. The device identifies thegesture performed and the orientation of the device relative to the userand corresponds it to that activity.

One of the feature modules can perform object identification based ontouch.

One of the feature modules can be a stimulation control module thatstimulates the user with an electrical current in order to elicit anaction based on a desired setting. Sending current through the devicewith a variable controllable resistor allows the device to be used forstimulating certain neuromuscular areas for therapeutic and othermedical purposes.

Calibration

A calibration process is a step in the use of one or more user interfacedevices such as the wrist band. The calibration can be physical orelectronic (e.g., digital) or a combination of them. Among the goals ofthe calibration are to improve user placement of, for example, the wristdevice (and in turn the electrodes on the wrist device) or to improvethe signal-to-noise ratio of the detected electrical potentials or both.In some cases, the user interface will provide on-screen instructions tohelp the user establish proper calibration of the device. This may alsoinclude a short testing phase to ensure activations are being recordedaccurately by the device.

For purposes of calibration, it can be useful to provide on a userinterface, visualizations of data values, characterizations, or athree-dimensional representation of a body part that moves in responseto such characterizations.

Data Sources

Much of the discussion to this point has related to the detection,identification, processing, and use of electrical potentials produced bynerve electrical signals in the wrist. The technology that we describemay be used with respect to nerve electrical signals in other parts ofthe body, including the brain and other extremities, for example, andcan be used for detection identification, processing, and use ofelectrical potentials produced by other tissues such as muscle tissue.In addition, a wide variety of other data sources can be used by theprocesser in generating characterizations of gestures, intent, andmuscle contractions, among others.

For example, combinations of data values from native infrared cameras,RGB cameras, and LED displays can be used to generate eye gaze data foruse in combination with electrical potential information for generatingcharacterizations.

In some implementations, electromyography (EMG) information can becaptured at the wrist, for example, along with the nerve electricalsignal information. The EMG information can be useful in its own rightand as data source to the processor for improving the quality of thedata values generated by the processor. The EMG information can becaptured by one or more secondary electrodes on a wrist device, forexample.

In some cases, other data sources such as IMUs can provide informationthat can be interpreted along with the data values from the processor todisambiguate gestures and direct interaction with the user interface ofa user interface device, for example, to distinguish a user making agesture in the air from a user touching a control of a user interface ofan application. In this sense, more broadly, combining data values fromthe processor with data from other sources such as IMUs can be helpfulin recognizing intent of a user.

There are pathways of electrical activity between the user's brain andthe parts of the body (for example, the fingers) that ultimately move asa result of intentions formulated in the brain. The pathways begin inthe brain (the site of neurological activity and associated electricalactivity that can be measured on the skull) and extend along the nerves(which also exhibit electrical activity along their lengths) to tissuesthat can include muscles and possibly tendons (where the contractions orrelaxations of muscles and associated electrical activity occur).Electrical activity occurs first in the brain and then along the nervesand the muscle contractions and relaxations then happen.

The speed of determining a person's intent and gestures may dependpartly on how near to the brain neurological activity measurements aremade. The accuracy and robustness of the determinations of intent andgesture may depend on the number and locations of places wheremeasurements are made. For example, although using IMUs to determine theactual motions of the fingers or hand may provide very good informationabout a gesture, that information is derived after a delay associatedwith the time required for the brain neurological activity, thepropagation of electrical signals along the nerves, the reaction of themuscles, and the processing of the IMU signals. In addition there may besubtle aspects of intent and gesture that cannot be fully determinedfrom IMU signals alone, but only by also measuring and processing nervesignals and brain signals.

In some implementations of the technologies that we describe,measurements are taken not only of nerve electrical activity (say, atthe wrist) and of actual motion of parts of the body (for example, usingIMUs to measure motion of fingers) but also of nerve electrical activityat places nearer to the brain (e.g., by measurements at the skin alongthe arm) and of brain electrical activity (by EEG or similartechniques). Analyzing the relative timing and intensities of electricalsignals and finger and hand motion along the pathways and thecombinations of these signals and motions can enable richer, morecomplex, and faster interpretation of intent and gesture, which can beuseful for many applications, including gaming, manufacturing, andcontrol of applications.

The technologies that we describe include platforms for the study of thepathways and for developing machine learning and other techniques forinterpreting the signals and motions along the pathways. In some cases,it may be possible, by such studies, to correlate brain electricalactivity sufficiently accurately to measured signals at the wrist andmeasured motions of the hand and fingers to enable reliance solely onthe brain electrical signals as good indicators of intent and gesture.

Applications

The applications for the technologies that we describe are wide rangingand include industrial, business, consumer, medical, institutional, andmilitary uses, for example. Functionally the applications span a broadrange, including monitoring, control, feedback, actuation,communication, and data accumulation and analysis, among others.

Monitoring applications involve detecting, storing, and accumulatingdata values for tissue electrical signals of individuals and groups andpopulations of individuals, such as monitoring nerve electrical signals(sometimes referred to as electroneurography or ENG signals), ENG orsENG (surface ENG) signals in combination with EMG or sEMG signals, andmonitoring other combinations of EMG, sEMG, ENG, sENG, EEG, ECG, and IMUsignals, for example.

Such monitoring can provide bodies of data useful in measuring oranalyzing the fitness (in terms of muscular strength or stamina, forexample) of groups of subjects or individual subjects, includingmilitary personnel. By also using data that represents muscle strength,the monitored bodies of electrical signal data can be used to developcorrelations between tissue electrical signals and muscle strength andthose correlations can be used to characterize muscle strength based ondata values derived from skin electrical potentials, for example.

Monitoring applications can be useful in medical fields. Among suchapplications are the monitoring of heart rate and EKG from the wrist orfoot. The heart generates strong electrical signals in the one millivoltrange that can be detected on the surface of the skin on the chest aswell as on the appendages. Heart rate signals occur at a frequency thatis discernable from other muscle and nerve signals. The importance oftracking heart rate is for determining measures of the user's health orcondition. This is also useful because traditional methods to captureheart behavior are possible only on the chest. These neuromuscularsensors that we have described can disambiguate ECG patterns on theentire body including the wrist and foot. Traditional wrist basedmethods using infrared technology measure only heart rate and not heartbehavior such as the timing of the heart chambers. The radial pulse ispalpable on the wrist. While pulse can also be measured on the dorsalispedis and posterior tibial areas of the foot and ankle respectively,among other locations on the body. This gives a multitude of methods andlocations to perform an EKG depending on the patient's or physician'sneeds.

Monitoring the state and progression of neurological disorders such asALS and other neurodegenerative disorders can also be useful. Moregenerally, any context in which non-invasive accurate monitoring ofnerve electrical signals would be useful for diagnosis or therapy orobservation of progress of a disease with or without therapy would besusceptible to useful applications. For example, observation oftherapeutic outcomes, such as after any surgery that involves orimplicates the condition or operation of nerves or muscle or othertissue, may be useful. Monitoring can also be used in diagnosing,quantifying or treating neuromuscular conditions; tracking epilepticactivity; monitoring to predict a future state, condition, or change innerves or muscles; or monitoring and analysis of muscle cramps.Monitoring can be useful in determining trends that are reflected intissue electrical signals, including muscle deterioration and diseaseprogress.

In some instances, patients can be provided with real time cues andfeedback to aid in proper muscle recruitment and movement patterns toimprove rehabilitation outcomes. Other applications include limb lossand spinal injury.

When monitoring occurs, the data values can be received, stored, oraccumulated not only at the user interface device (such as in the memoryof the wrist device) but also in devices (such as smart phones) that arein the vicinity of the user interface device, and more generally in oneor more components of the HMI platform, or in one or more of theapplications. In some implementations, the data values from one or moresubjects, or groups, or populations can be received, stored,accumulated, processed, or used on one or more servers 920 operatedcentrally (e.g., in the “cloud”) and the data values or the results ofsuch activities can be made available from the cloud to users 922through web browsers 924 anywhere. For example, real time or archiveddata values or characterizations can be used by clinicians to makedecisions about and study diagnosis, prognosis, and therapy. Patientdata values can be collected and stored locally or in the cloud.Functional changes in muscles or nerves, for example, can be monitoredand used in determining required modification of configurations orassistive technology setup and in controlling remote configuration ofassistive technology.

Monitoring can be used for applications in which muscle contractions,gestures, or intent are tracked and mapped to a physics-based model ofthe motion of the hand or another part of a body. The use of thephysics-based model can enhance the quality of the characterizationsderived by the processor by enabling correction of the characterizationsbased on constraints implied by the models. For instance, if someoneextends his finger, the model constraints wouldn't allow this gesturecharacterization to move the finger beyond that which is capable of thejoint and musculature connected to the finger.

Another broad area of applications involves control of a device orprocess based on the data values or characterizations (e.g., gestures orintent) derived by the technology.

In a simple but useful application, control can involve usingcharacterization information to control an onscreen avatar using fingermotions.

The control can be control of mobile devices, for example. In themedical realm, control can enable a patient to use gestures and intentto control the operation or state of a mobility device, includingcontrol of a car, a wheelchair, personal robotics, controllableorthoses.

Control can include control of a household environment or a component orelement of such an environment, such as elevators, windows, doors,curtains, blinds, thermostats, heating and air conditioning devices,electrical sockets, intercoms, door lock releases, sump pumps, lights,alarms and other security devices, media devices, cleaning and cookingdevices, bathing and other bathroom equipment, entertainment devicesincluding televisions, media centers, video games, game devices, DVDs,telephone and other communication equipment, and emergency callrecordings to be played back on a telephone, among others.

In some implementations, control can be control of devices in contextsof safety, security, silence, privacy, policing, and intelligence.Gestures or intent of a user derived as one or more characterizationsbased on detected electrical potentials could be used, for example, totrigger a silent alarm for help or to alert another party of a danger;to trigger an emergency call, such as by an avalanche victim, by a bankemployee in the context of a robbery, or by military or police personnelto control military or policing devices silently or remotely.

Control applications include those for controlling functions of gamecontrollers, joysticks, or other virtual reality control devicesincluding augmented reality glasses, for example, in an industrialsetting. Other control applications include navigating menu selectionsof a user interface. In some cases, a user could be holding something inher hands and also be using gestures to control an interface beingpresented by augmented reality glasses. In some cases, a person could bewalking along a course indicated on a map in augmented reality glassesand could use gestures to control the interface. In some applicationscontrol could be asserted by gestures within a space suit.

In general, the technology can be especially useful in situations inwhich the ability of the user to perform full gestures is constrained,such as in a space suit, or a SCUBA suit, or when the user's hands orother parts of his body are occupied in other ways and therefore cannotbe used to perform normal gestures.

In some instances, a user could control a virtual reality or augmentedreality avatar.

A substantial range of applications involve interaction between userinterface devices (such as a wrist device) and features of applicationssuch as smart glasses. For example, the HMI can pass characterizationinformation to the smart glasses (or other head-worn displays) toimplement hands-free, voice-free control. For instance, augmentedreality systems for pharmaceutical clean rooms can be provided withcharacterizations of user gestures or intent to control production,quality control, packaging, and other functions in the manufacturing ofpharmaceuticals. In such contexts, users often are performing complextasks and do not have free hands to control the augmented realitydisplay. In addition, such environments may be noisy or multiple peopleare present, making voice control not viable.

In some implementations, for example, as shown in FIG. 18 , userinterface devices (such as a wrist device) 950 can detect motions of thehand 952, wrist 954, or fingers 956, for example, made by a user toexpress gestures (or intent). An HMI 958 can include a translationcomponent 961 that executes an algorithm to translate the detectedmotions into control of one or more displayed elements 960 of adisplayed user interface 962 of an application 964.

Among the motions of the hand, wrist, or fingers that could be detectedare (in aviation terms) yawing or pitching or rolling of the hand orwrist or both or combinations of yawing, pitching, or rolling or avariety of other motions of the hand, wrist, or fingers (together calledthe “hand/wrist/finger motions”). Among other things, the user can twistor rotate 966 his hand (often in combination with twisting or rotationof his wrist) about an axis 968 generally aligned with the length of theuser's forearm. The hand/wrist/finger motions can be interpreted as oneor more actions, gestures, or intentions meant to control the displayedelements 960. The displayed elements could be a cursor, or an insertionpoint, or a graphical element such as a sphere, or a character such as acartoon figure or an avatar.

In some cases, the hand/wrist/finger motions could include twisting orrotation of the hand and wrist, the displayed element could include asphere or some representation analogous to a sphere 960, and thetwisting or rotation can be translated to a corresponding rotation orother motion of the sphere relative to the displayed user interface. Bymanipulating the sphere a user can in effect use the sphere forinteracting with the displayed user interface, for example, to point toother displayed elements, move the elements, select actions, performactions, and other activities. More generally, the hand/wrist/fingermotions can be used to control any controllable aspect of an applicationthrough its displayed user interface, such as menus, content entry,gaming elements, interaction with other users, or navigation through atwo-dimensional or three-dimensional displayed space, among otherthings.

The translation component 961 of the HMI 958 can include a mappingalgorithm 968 that translates signals 970 representing thehand/wrist/finger motions to corresponding directions, orientations,rates of motion, or magnitude of motion (or combinations of them) of thedisplayed element, enabling the user to use gestures to control thedisplayed element.

In some examples, the direction, orientation, rate, and magnitude oftwisting or rotation of the user's hand can be translated intoproportional motion of the sphere or other displayed element havingcorresponding direction, orientation, rate of motion, and magnitude. Insome cases, one or more of the direction, orientation, rate of motion,and extent of motion of the displayed element can be other than a directproportional mapping from the hand/wrist/finger motions. The mapping canhave any degree of simplicity or complexity.

One useful mapping algorithm can translate the magnitude of motion ofthe hand and wrist into a proportionately larger magnitude of motion ofthe sphere or other displayed element. For example, a 5 degree rotationof the hand and wrist could translate to a 20 degree rotation of thesphere, representing an amplification factor of the algorithm. Theamplification factor could change dynamically with one or more featuresof the hand/wrist/finger motions such as magnitude or acceleration orboth. For example, a small, fast rotation of the hand and wrist could betranslated to a large rotation of the sphere. Conversely, slowacceleration of the hand and wrist could be translated using arelatively low amplification factor.

In some examples, motions of the wrist including rotation or panning orboth could be used to control a rotary dial to choose among a set oficons arranged at respective angular positions relative to the rotarydial. The wrist motions could be translated one-to-one such that a45-degree rotation translates to a corresponding 45-degree rotation in arotary or linear array. Or small motions could translate to largermotions through amplification as noted above.

Communication between the wrist device and smart glasses can also beused for application switching (e.g., object oriented applicationswitching).

With respect to feedback, the technologies that we have described haveapplications in contexts in which providing feedback about detectedelectrical potentials, or characterizations derived from them (e.g.,muscle contractions, gestures, intent), can be useful for improvingconduct, behavior, accuracy, or precision of gestures or intent ormuscle contraction. Feedback could include visual, audible, haptic, orother modes of presentation to a user. Feedback could occur in real timeor later. The feedback can directly present (say visually or audibly) anactual gesture, intent, or muscle contraction back to the user in realtime to enable the user to immediately and directly alter an attemptedgesture, intent, or muscle contraction to tend toward an ideal. In somecases, the feedback can present information that indirectly implies tothe user how changes in attempted muscle contractions, gestures, orintent could more accurately, precisely, or usefully serve a goal orpurpose.

Haptic feedback can be presented to the sense of touch of the user, forexample, by vibrating or otherwise moving a smart phone or a wristdevice or another device with respect to the skin of the user. Hapticfeedback can improve control or triggering actions of a user. In somecases, haptic feedback can help in retraining muscles in patients withparesis (by improving neuroplasticity). To accomplish this, patientswould be provided with a physical therapy activity or regime where thedevice would recognize the completion of the action or gestureresponding with a haptic cue that it was performed correctly. It thenbecomes a training exercise with very clear and prompt cues based on theuser's own interactions with the device.

In some instances, haptic feedback could provide cues as a non-visible,non-auditory interface to enable a user to, for example, navigatethrough menus of an application using a haptic vocabulary. Moregenerally, haptic feedback can use a haptic vocabulary (in which theuser can be trained) to provide a simple, discreet, non-visible,non-auditory technique for informing a user about a state or conditionor change in a process, device, or application that can be achieved by acorresponding muscle contraction, gesture, or intent of the user thatyields detectable tissue electrical signals.

In some cases, feedback can be visual using lights, displays, or othervisual devices and cues. For example, the feedback can be used toprovide visualization cues about actual electrical potentials, datavalues, muscle contractions or relaxations, gestures, or intent. Thevisual devices and cues can be presented on a user interface of a smartwatch or wrist device, on a smart phone, or on an application such assmart glasses.

Feedback can be of any kind that aids actuation or control of one ofmore features of a virtual reality or augmented reality or an interfaceor device through which a virtual reality or augmented reality ispresented. For example, a user can move his fingers using the wristdevice described above to control a drone moving through a virtualreality world or through the real world. Thus, the feedback can beuseful in military applications in which augmented or virtual realitiesare being presented, or in a medical context in which a display screenpresents an augmented reality view of a hidden site at which a procedureis being performed.

In general, feedback can provide contextual, intuitive,application-specific switching and control of application devices andprocesses.

Some applications of the technologies are in the broad field ofcommunication and enable users to communicate based on musclecontractions and relaxations, gestures, intent, or othercharacterizations. For example, such a user can cause suchcharacterizations to cause typing of characters into a word processingapplication, text into a texting application, moving a mouse, engagingin alternative or augmentative communication, sending Morse code, andcommunicating in multiple channels in parallel, e.g., by both talkingand moving fingers, or using eye movements and hand movements at thesame time through distinct communication channels. In covertapplications, control or communication could be done by a user bygestures of his hand in his pocket, hidden from view. Hidden gesturescan be used for phone control, clandestine communication, prostheticsand exoskeletons, or keyboard or mouse replacement.

Some applications can involve receiving, collecting, and analyzing datavalues generated by the technology. For example, in implementations thatuse machine learning techniques for deriving characterizations from datavalues, it can be useful to collect and analyze data values for a singleuser as a means, for example, of training a classifier to classifyfuture data values based on feature sets among various gestures orintent. Collecting and analyzing data values for groups or populationsof thousands or millions of users can enable the generation of usefultemplates (feature sets) for later classification for any user. In someapplications, data values combined with knowledge of the correspondingorientation and position of the user interface device, such as a wristdevice, enables determining advantageous orientations and positions forthe electrodes of other user interface devices, for example,orientations and positions at which tissue electrical signals areparticularly robust or can be effectively detected and processed.

As mentioned above, some applications involve control of andcommunication with smart glasses. The gesture vocabulary for controllingor communicating with smart glasses could include: (1) entering into a“listen” mode by moving two fingers at the same time (2) subsequentlyrotating the wrist to navigate through a circular selection menu of,say, three or more items. (3) subsequently moving fingers to make theselection or releasing the held “listen” movement to make the selection.Typing into the user interface can be done using gestures to navigatethrough groupings of letters, for example.

In general, in applying the technology, a vocabulary ofcharacterizations can be established and correlated with particularfeature sets of the data values. Once a vocabulary has been established,the vocabulary can be used for a wide variety of purposes with respectto applications, just as any language vocabulary can be used. Thevocabulary can be altered, updated, and enhanced as needed both bydevelopers and by users. The correlation between data values andelements of the vocabulary can be updated, enhanced, and improved. TheHMI platform or the user interface device or both can maintain andimplement such a vocabulary and serve in effect as a translator betweendata values and elements of the vocabulary; then applications can usecommunications that are phrased according to the vocabulary in a widevariety of ways. Such a vocabulary can be implemented silently withoutrequiring speech or (in some cases) actual motion of any part of theuser's body. In this way, a user's intentions can be used to expresscommunications based on the vocabulary. Such a vocabulary can bespecific to a user or to a group or to a general population.

The text below (and in some cases earlier) includes certain text orinformation from the provisional application (U.S. patent application62/429,334, filed Dec. 2, 2016) with respect to which this applicationis entitled to the priority of the filing date. (The entire provisionalapplication is incorporated here by reference.)

The capability of measuring electrical signals in nerves has varioususes, for example, helping people with Amyotrophic Lateral Sclerosis(ALS), ALS is a disease in which the neurons that control voluntarymuscles degenerate and die. Its causes are not fully known orunderstood, although some incidents are attributed to genetics. A personwith ALS will experience muscle stiffness, cramping, twitching, andweakness as neurons are lost and the muscles atrophy. Communicationbecomes difficult. Eventually, the person may lose muscle control neededfor speech, for eating, and even for breathing.

Surface Electromyography (sEMG) is a form of electromyography thatmeasures electrical activity in a muscle using sensors (electrodes)placed on the skin surface above the muscle. Two or more such electrodescan be used to measure electric potential (voltage) differences betweendistinct electrodes. Using these electrodes, an electromyograph candetect the electric potential generated by the muscle cells whenactivated by neurological (or electrical) signals. sEMG is less invasivethan other forms of electromyography that require subcutaneous sensors.

Electroneurograms (ENGs) provide visualizations of electrical activityof neurons based on electrical signals obtained directly from electrodesplaced into the neuron tissue. The electrical signals provided by theelectrodes capture action potentials, which are potentials that rise andfall quickly. For example, when the brain sends a signal through a nerveto neurons in the vicinity of a muscle, the signal typically takes theform of an action potential.

Some rudimentary solutions exist to help a person with AmyotrophicLateral Sclerosis (ALS) communicate with others, such as tracking eyegaze, but these solutions can be exhausting and difficult to use. Forexample, only about 10% of people with ALS can even use the eye gazemethod. As described herein, surface electromyography (sEMG) can be usedto control electrical equipment such as wheelchairs, remote controldoors, entertainment systems (e.g., radios and televisions), andcomputer systems. This new level of control can more effectively andefficiently facilitate communication than existing technologies and willimprove the over-all quality of life for persons with ALS.

A person with healthy muscles has surface electromyography (sEMG)measurable signals around 1 to 2 millivolts (i.e., 0.001 to 0.002volts). People with Amyotrophic Lateral Sclerosis (ALS) can havecompromised nerves and atrophied muscles such that the strength of theirmeasurable signals could be in the 5-10 microvolt range, two hundredtimes smaller than a person with healthy muscles. Existing technologythat uses sEMG, e.g., prosthetics control, require healthy muscle andnerve signals in the millivolt range. Accordingly, these technologiesdon't translate well to people with ALS. However, even if a person withALS has no ability to move, the residual nerves in their forearm, leg,and jaw may still generate electrical signals that can be detected withsEMG.

As described here, sEMG sensors are used to detect a person's nerve andmuscle signals and used to interface with electronic devices, e.g.,computers and computer-controlled equipment. Configurations arecustomizable and modular to accommodate the range of user abilities andpreferences. Electromyography activation is painless, requires zeromovement, and is adaptable to ALS progression. Compared to eye gaze,which less than 10% of people with ALS use due to physical and mentalstrain, these systems are designed for people with little or no movementability in their entire body. Further, because some implementations donot require any physical movement or gestures by a user, a person withALS can use these solutions even if they are fully immobilized.

FIG. 1 is a diagram of an example sensor hub 130 in communication withsEMG sensors 140 and interfacing with a client device 120 in anoperation environment 100. In broad overview of the operationenvironment 100, a user 122 wears one or more sensor 140. Each sensor140 includes one or more electrode patches 142 and a transmitter 146.The sensor transmitters 146 transmit data to a sensor hub 130. Thesensor hub 130 includes a receiver 136 for receiving the datatransmitted from the sensors 140. The sensor hub 130 includes a clientinterface 132, an analytics interface 135, a sensor controller 137, andmemory 138. The sensor hub 130 includes a processing stack 170 thatprocesses data received from the sensors 140 by amplifying and filteringreceived data. The illustrated processing stack 170 includes apre-amplifier 172, a filter 174, and a gain-amplifier 176. The sensorhub 130 interacts with a client device 120 via the client interface 132.In some implementations, the sensor hub 130 sends data to an analyticsserver 150 via the analytics interface 135. The analytics interface 135exchanges data with the analytics server 150 via a network 110. In someimplementations, the client interface 132 also exchanges data with theclient device 120 via the network 110.

Referring to FIG. 1 in more detail, each sEMG sensor 140 includes one ormore electrode patches 142 and a transmitter 146. The sensors 140 alsoinclude a power source such as a rechargeable battery (not shown). AnsEMG sensor 140 detects nerve activation in a person (e.g., a personwith ALS), and provides data (e.g., a signal representative of a levelof nerve activation) to the sensor hub 130. In some implementations, thesensor hub 130 is incorporated into a client device 120, such that thesensors 140 provide this data directly to the client device 120. In someimplementations, multiple sEMG sensors 140 may be placed on a person,e.g., along a nerve path such as at the top of a forearm, middle of theforearm, and bottom of the forearm. In some implementations, the sEMGsensors 140 placed along a nerve path with detect the same activationevent at slightly different moments in time; in some suchimplementations, the data is time shifted to align the signals such thatthey are coincident. This can help boost an activation signal above anoise level.

Each sEMG sensor 140 includes electrode patches 142 of electrodes thatare placed either directly on the skin or very close to the skin, e.g.,separated only by a thin film or layer of fabric (e.g., a conductivefabric). In some implementations, the sensors 140 are replaceable padsdesigned to adhere to the user 122 for a few hours, days, weeks, ormonths. In some implementations, the sensors 140 fit easily underneathclothing. In some implementations, the sensors 140 are wearable itemswhere the components (e.g., the electrode patches 142 and transmitter146) are packaged into straps, arm bands, watch bands, sweat bands, headbands, shirts, shorts, socks, or other items that can be worn on thebody. In some implementations, wearable sensors 140 are designed forall-day (“24/7”) wearability. Wearable sensors 140 may be designed foruse in bed, in a wheelchair, outdoors, in a vehicle, and so forth. Insome implementations, a wearable sensor 140 designed for use in bed canserve as an emergency alarm during sleep. The electrode patches 142 areoriented by the sEMG sensor 140 housing to place electrodes in directcontact with the skin of the user 122, or in near-direct contact withthe skin, e.g., separated from the skin by only a thin film or layer offabric. Each sEMG sensor 140 includes one or more electrode patches 142,and each electrode patch 142 includes one or more electrodes. To measurean electric potential (voltage) difference, at least two electrodes areneeded. Medical grade electrodes are used. In some implementations, anelectrode can be used for five or more days. In some implementations,the electrode patch 142 can be replaced in the sensor 140 withoutdiscarding the transmitter 146 or other components of the sensor 140.

In some implementations, each sEMG sensor 140 includes additionalcontextual and environmental monitoring elements such as thermometers,accelerometers, gyroscopes, barometers, and moisture or humiditysensors. Information from such elements is transmitted to the sensor hub130 in the same manner as described with reference to the sEMGinformation from the electrode patches 142. The contextual andenvironmental monitoring information can be used to differentiate signalnoise for nerve activation signals. Temperature shifts, humidity shifts,incidental movement, and other such contextual or environmental changesmay be accommodated, for example, by adjusting how information from thesensor 140 is handled at the sensor hub 130.

Each sEMG sensor 140 includes a transmitter 146 to transmit data to asensor hub 130. In some implementations, the transmitters 146 transmitan analog signal representative of the electric potential (voltage)difference measured from the electrodes in the electrode patches 142. Insome implementations, the analog signal is amplified. In someimplementations, the analog signal is broadcast using a frequencymodulation or amplitude modulation of preconfigured radio frequencies.In some implementations, the transmitter encodes the analog signal intoa digital signal, e.g., using an analog to digital (ATD) converter. Insome such implementations, the digital signal is transmitted by thetransmitters 146 in packet form, e.g., using a personal area network(PAN) protocol such as ANT+, BLUETOOTH, or ZIGBEE. Although illustratedin FIG. 1 as wireless, in some implementations, a sEMG sensor 140 doesnot communicate with the sensor hub 130 by radio and is, instead,configured to transmit data to the sensor hub 130 via a wired linkage.

The sensor hub 130 includes a receiver 136. The receiver 136 is anelectrical circuit for receiving the data transmitted from the sensors140. In some implementations, the receiver 136 is a radio receiver. Insome implementations, the receiver 136 implements a personal areanetwork (PAN) protocol such as ANT+, BLUETOOTH, or ZIGBEE. In someimplementations, the receiver 136 is configured to receive signals frommultiple sensor transmitters 146 concurrently. In some implementations,the receiver 136 is configured to receive signals from multiple sensortransmitters 146 using a time division multiplexing approach. In someimplementations, the sensor hub 130 includes multiple receivers 136.

The sensor hub 130 includes a processing stack 170 that processes datareceived by the receiver 136 from the sensors 140. The processing stack170 amplifies and filters received data. The illustrated processingstack 170 includes a pre-amplifier 172 (“pre-amp”), a filter 174, and again-amplifier 176 (“gain-amp”). In some implementations, thepre-amplifier 172 is a programmable operational high-gain electronicvoltage amplifier (“op-amp”) preamplifier. In some implementations, thefinite bandwidth of the pre-amplifier 172 is configured to emphasizebandwidths specific to nerve activation signal levels of a person withALS (in some implementations, this is an average range for people withALS; in some implementations, this is tuned to a specific individual,e.g., an ALS patient). In some implementations, the pre-amplifier 172can dynamically select thresholds above a noise floor. The amplifiedsignal from the pre-amplifier 172 is then passed through one or morefilters 174.

The filter 174 is a signal processing filter used to improve separationof input signals from the sensors 140 from general electronic noise. Insome implementations, the filter 174 is an analog filter (e.g., an RLCfilter); in some implementations, the filter 174 is a digital filter. Insome implementations, the filter 174 is a high-pass filter, attenuatingsignals below a noise floor. In some implementations, the filter 174 isa band-pass filter tuned to a specific signal range. In someimplementations, the filter 174 is configured to attenuate signals belowand/or above a range specific to nerve activation signal levels of aperson with ALS (in some implementations, this is an average range forpeople with ALS; in some implementations, this is tuned to a specificindividual, e.g., an ALS patient). For example, a digital high-passfilter or digital band-pass filter can be tuned in this manner.

A gain-amplifier 176 boosts the strength of the filtered signal receivedfrom the filter 174. In some implementations, the gain-amplifier 176 isomitted. In some implementations, the processing stack 170 includes ananalog-to-digital (ATD) converter that converts an analog signal fromthe gain-amplifier 176 (or directly from a filter 174) into a digitalsignal.

In some alternative implementations, some or all elements of theprocessing stack 170 are packaged in the sensor 146 and the outputtherefrom is transmitted to the receiver 136. Such alternativeimplementations are not shown in FIG. 1 , but are considered to bewithin the scope of this disclosure.

The processing stack 170 provides the results of amplifying andfiltering the received sensor input to the client device 120 via aclient interface 132. In some implementations, the client interface 132exchanges control messages with a control application executed at theclient device 120. In some such implementations, the control applicationinterprets the sensor data and converts the sensor data into controlinstructions at the client device 120. In some implementations, theclient interface 132 interprets the sensor data and converts the sensordata into control instructions at the sensor hub 130. The instructionsare then sent by the sensor hub 130 to the client device 120, e.g., viaa physical or wireless network link.

The sensor hub 130 interacts with a client device 120 via the clientinterface 132. In some implementations, the sensor hub 130 sends data toan analytics server 150 via the analytics interface 135. The analyticsinterface 135 exchanges data with the analytics server 150 via a network110. In some implementations, the client interface 132 also exchangesdata with the client device 120 via the network 110. For example, insome implementations, the client interface 132 implements a BLUETOOTHcommunication channel with the client device 120.

In some implementations, the sensor hub 130 collects analyticsinformation, e.g., information that may be used to improve the tuning ofthe sensor equipment and the processing stack 170. Analytics informationis stored in memory 138 and/or transmitted to an analytics server 150over a network 110. Analytics information may include, for example,measurements of metrics for daily usage, strength of signals for variousnerve and muscle activations, and user patterns (which may be anonymizedto protect patient privacy). All information is managed in compliancewith the Health Insurance Portability and Accountability Act (HIPAA) andpatient information is always protected. In some implementations, theanalytics interface 135 exchanges messages with the analytics server 150over a network 110, e.g., using a custom protocol layered over TCP. Insome implementations, the analytics server 150 stores calibration andconfiguration information associated with a particular user 122. Thisallows the user 122 to use the same calibration and configurationinformation while using multiple implementations, e.g., at a diagnosticclinic, at home, at a hospital, and at other therapeutic centers. Insome implementations, the analytics interface 135 receives updates fromthe analytics server 150. Updates may include, for example, refinementsfor tuning elements of the processing stack 170. This allows each sensorhub 130 to benefit from aggregate analysis of data from sEMG sensors 140placed on a variety of people with ALS.

Still referring to FIG. 1 , the memory 138 provides data storage forpersisting data in the sensor hub 130. The memory 138 may be implementedusing one or more data storage devices. The data storage devices may beany memory device suitable for storing computer readable data. The datastorage devices may include a device with fixed storage or a device forreading removable storage media. Examples include all forms ofnon-volatile memory, media and memory devices, semiconductor memorydevices (e.g., EPROM, EEPROM, SDRAM, and flash memory devices), magneticdisks, magneto optical disks, and optical discs (e.g., CD ROM, DVD-ROM,or BLU-RAY discs). Example implementations of suitable data storagedevices include storage area networks (SAN), network attached storage(NAS), and redundant storage arrays.

The network 110 enables communication between network-connected devices.In some implementations, data flows through the network 110 from asource node to a destination node as a flow of data packets, e.g., inthe form of data packets in accordance with the Open SystemsInterconnection (OSI) layers. A flow of packets may use, for example, anOSI layer-4 transport protocol such as the User Datagram Protocol (UDP),the Transmission Control Protocol (TCP), or the Stream ControlTransmission Protocol (SCTP), transmitted via the network 110 layeredover an OSI layer-3 network protocol such as Internet Protocol (IP),e.g., IPv4 or IPv6. The network 110 is composed of various networkdevices linked together to form one or more communication paths betweenparticipating devices. Each networked device includes at least onenetwork interface for receiving and/or transmitting data, typically asone or more data packets. An illustrative network 110 is the Internet;however, other networks may be used. The network 110 may be composed ofmultiple connected sub-networks. The network 110 can be a local-areanetwork (LAN) such as a company intranet, a metropolitan area network(MAN), a wide area network (WAN), an inter network such as the Internet,or a peer-to-peer network, e.g., an ad hoc WiFi peer-to-peer network.The data links between devices in the network 110 may be any combinationof wired links (e.g., fiber optic, mesh, coaxial, twisted-pair such asCat-5 or Cat-6, etc.) and/or wireless links (e.g., radio, satellite,microwave, etc.). The network 110 may include carrier networks formobile communication devices, e.g., networks implementing wirelesscommunication protocols such as the Global System for MobileCommunications (GSM), Code Division Multiple Access (CDMA), TimeDivision Synchronous Code Division Multiple Access (TD SCDMA), Long-TermEvolution (LTE), or any other such protocol including, but not limitedto, so-called generation “3G,” “4G,” and “5G” protocols. The network mayinclude short-range wireless links, e.g., via Wi-Fi, ANT+, BLUETOOTH, orZIGBEE, sometimes referred to as a personal area network (PAN) or meshnetwork. The network 110 may be public, private, or a combination ofpublic and private networks. The network 110 may be any type and/or formof data network and/or communication network.

The example client device 120 is a computing system, or processor-baseddevice, that executes applications, presents output to a user, andreceives input from the user. The client device 120 is capable ofreceiving information from the sensor hub 130, e.g., via a direct wiredlink, via a wireless link (e.g., on a PAN), via an infrared link, via anetwork 110, or by other communication means. In some implementations,the client device 120 can be controlled by the sensor hub 130. Theclient device 120 may be any kind of computing device, including, forexample, a desktop computer, a laptop or notepad computer, a mobiledevice such as a tablet or electronic pad, a personal digital assistant,a smart phone, a video phone, or an entertainment device such as aradio, a television, or any other such device. The client device 120 maybe a robotic device such as a wheelchair, a remote control door, etc. Insome implementations, the client device 120 includes one or morehardware elements for facilitating data input and data presentation,e.g., a keyboard, a display, a touch screen, a microphone, a speaker,and/or a haptic feedback device. In some implementations, the clientdevice 120 is implemented using special purpose logic circuitry, e.g.,an application specific integrated circuit (ASIC). In someimplementations, the client device 120 is implemented using a system ona chip (SoC) semiconductor device that includes at least one processor(or microprocessor) core. In some implementations, the client device 120is implemented using a general-purpose computing processor. FIG. 8 ,described in more detail below, illustrates a computing device that, insome configurations, is suitable for use as a client device 120.

FIG. 2 is a flowchart for an example method 200 of monitoring sEMGsensors. This description refers to a single sensor, but is applicableto one or more sensors, including groups or sets of sensors. Thisdescription allocates all monitoring and control tasks to the sensorcontroller 137; however, in some implementations, some or all of thesetasks may be performed by the client interface 132 or by some otherelement in the sensor hub 130, on a sensor 140, or even at the clientdevice 120. In broad overview, at stage 210, the sensor controller 137calibrates a sensor 140. Calibration may be automated, e.g., bymeasuring sensor noise to establish a baseline. At stage 220, the sensorcontroller 137 monitors the sensor 140 for signals and, at stage 230,detects a signal from the sensor 140. If the sensor is part of amulti-sensor set (branch 240), then at stage 250, the sensor controller137 compares sensor measurements from the detected signal to data fromother sensors in the multi-sensor set. A true nerve activation signalmay generate signals at multiple sensors in the set at various strengthsand slightly different times (e.g., rippling through the sensors) suchthat analysis of the aggregate conditions can differentiate between thetrue nerve activation signal and general system noise. At stage 260, thesensor controller 137 determines if the detected signal satisfies actioncriteria. For example, determining whether the signal (or set ofsignals) exceeds a noise threshold or is sustained for a minimum lengthof time. If so, then at stage 270 the sensor controller 137 reports thesignal to the client interface 132. The client interface 132 can thenissue a control message to the client device 120, or report the signalto the client device 120, which can then determine the appropriateaction.

Referring to FIG. 2 in more detail, at stage 210, the sensor controller137 calibrates a sensor 140. Calibration may be automated, e.g., bymeasuring sensor noise to establish a baseline. In some implementations,a user 122 may wear the sensor 140 and trigger (or have an aide trigger)the calibration process while remaining passive. The sensor 140 gathersinformation about the baseline passive state, e.g., sEMG levels andcontextual noise information. In some implementations, calibration isfully automatic. In some implementations, calibration includes manualadjustments or configurations. For example, the user 122 may be asked toactivate various nerves and the sensor 140 can be further calibratedbased on the sensitivity needed to detect the nerve activations. In someimplementations, calibration is occasionally or periodically repeated.

At stage 220, the sensor controller 137 monitors the sensor 140 forsignals. In some implementations, the sensor 140 is constantly (orperiodically) reporting sensor status information. The sensor hub 130receives this status information from the sensor 140 and monitors it foractionable conditions.

At stage 230, the sensor controller 137 detects a signal from the sensor140. When a signal arrives from the sensor 140 that exceeds thebaseline, the sensor controller 137 attempts to classify the signal aseither intermittent noise or an actionable signal.

If the sensor is part of a multi-sensor set (branch 240), then at stage250, the sensor controller 137 compares sensor measurements from thedetected signal to data from other sensors in the multi-sensor set. Atrue nerve activation signal may generate signals at multiple sensors inthe set at various strengths and slightly different times (e.g.,rippling through the sensors) such that analysis of the aggregateconditions can differentiate between the true nerve activation signaland general system noise.

At stage 260, the sensor controller 137 determines if the detectedsignal satisfies action criteria. For example, determining whether thesignal (or set of signals) exceeds a noise threshold or is sustained fora minimum length of time.

If the sensor controller 137 determines that the detected signalsatisfies the action criteria, at stage 270, the sensor controller 137reports the signal to the client interface 132. The client interface 132can then issue a control message to the client device 120, or report thesignal to the client device 120, which can then determine theappropriate action.

FIG. 3A, FIG. 3B, and FIG. 3C are screen images for an exampleimplementation of a sensor monitor and interface. Flowcharts forinteractions shown in FIG. 3A, FIG. 3B, and FIG. 3C are described inmore detail below.

In broad overview, FIG. 3A depicts selection of an icon using a pointeror mouse-based interaction. As will be described, a vertical bar 310 anda horizontal bar 320 are manipulated by repeated input signals. Thevertical bar 310 passes through a selected coordinate on a display's Xaxis and the horizontal bar 320 passes through a selected coordinate onthe display's Y axis. The vertical bar 310 and horizontal bar 320 aremanipulated to intersect at the selected X and Y coordinates. Forexample, as described below in reference to FIG. 5 , the vertical bar310 can be moved horizontally in a horizontal scan and the horizontalbar 320 can be moved vertically in a vertical scan. An icon 330 at theresulting [X,Y] intersection can then be selected, e.g., to open acorresponding application.

FIG. 3B depicts text entry into an input field 340 using a virtualkeyboard 350. In some implementations, the virtual keyboard 350 is areproduction a numeric or telephone key layout. In some implementations,the virtual keyboard 350 is a set of directional arrows. In someimplementations, the virtual keyboard 350 is a reproduction of astandard “QWERTY” keyboard layout. In some implementations, the virtualkeyboard 350 uses an alphabetical layout. In some implementations, thevirtual keyboard 350 uses an alternative layout such as Dvorak orColemak. In some implementations, the virtual keyboard 350 is fullycustomizable such that a user or operator can configure the virtualkeyboard 350 for a particular user or use scenario. In someimplementations, the virtual keyboard 350 includes buttons for custommacros or character sets. For example, buttons may be presented in thevirtual keyboard 350 for emojis, common emoticons, or word suggestions.In some implementations, the virtual keyboard includes predictivebuttons populated with word completion predictions, next wordpredictions, and/or frequently used words or phrases.

FIG. 3C depicts a row of virtual keys 360 (from the virtual keyboard350) highlighted during a key selection process. A chart 370 provides avisualization of the signal strength for a sensor (or set of sensors) inuse to select from the row of virtual keys 360. The screen images forthe example implementation depicted in FIG. 3A, FIG. 3B, and FIG. 3C arereferenced in the following flowchart descriptions. The examples shownin FIG. 3A, FIG. 3B, and FIG. 3C are from just one of many possibleimplementations, and are not meant to be limiting.

FIG. 4 is a flowchart for an example method 400 of selecting a matrixfield using sEMG sensors. This description refers to a single sensor,but is applicable to one or more sensors, including groups or sets ofsensors. This description allocates all tasks to the client interface132; however, in some implementations, some or all of these tasks may beperformed by some other element in the sensor hub 130, on a sensor 140,or even at the client device 120. The method 400 may be used to selectfrom any matrix. For example, a key on a virtual keyboard (e.g., thevirtual keyboard 350 shown in FIG. 3B) can be selected using the method400. This description of FIG. 4 scans rows of the matrix before scanningcolumns of the matrix. In some implementations, columns of the matrixare scanned before rows of the matrix. In some implementations, a firstset of sensors is used to scan rows and a separate set of sensors isused to scan columns; whichever set of sensors is triggered determineswhether a row of the matrix is scanned or a column of the matrix isscanned. See, for example, FIG. 7 and the corresponding descriptionbelow. These implementation variations and others are within the scopeof this disclosure.

In broad overview of FIG. 4 , at stage 410, the client interface 132performs initialization functions to prepare for input selection. Forexample, the client interface 132 may request sensor recalibration, maypresent a selection matrix (e.g., a virtual keyboard) on a presentationscreen (e.g., at the client device 120), and begin monitoring selectionsensors. At stage 420, the client interface 132 receives notification ofa signal from a sensor 140 and determines if the signal is sufficient totake action (see, e.g., method 200 illustrated in FIG. 2 ). If not(e.g., if the signal is too short), the client interface 132 returns toinitialization stage 410 to wait for another signal. Otherwise, if thesignal is actionable, then at stage 430 the client interface 132 beginsscanning rows of the selection matrix. The client interface 132continues to scan rows of the selection matrix until signaled to stop.The last row scanned in stage 430 is identified as a selected row. Atstage 440, the client interface 132 receives notification of a signalfrom a sensor 140 and determines if the signal is sufficient totransition to a next action. If not (e.g., if the signal is too short),the client interface 132 returns to initialization stage 410 to restartthe method 400 and wait for another signal. At stage 440, the clientinterface 132 may also determine to return to stage 410 if too much timepasses since starting stage 430. If, at stage 440, an actionable signalhas been received, then at stage 450 the client interface 132 beginsscanning columns of the selection matrix. The client interface 132continues to scan columns of the selection matrix until signaled tostop. The last column scanned in stage 450 is identified as a selectedcolumn. At stage 460, the client interface 132 receives notification ofa signal from a sensor 140 and determines if the signal is sufficient totransition to a next action. If not (e.g., if the signal is too short),the client interface 132 returns to scanning rows at stage 430 andwaiting for another signal. At stage 460, the client interface 132 mayalso determine to return to stage 430 if too much time passes sincestarting stage 450. If, at stage 460, an actionable signal has beenreceived, then at stage 470 the client interface 132 selects the matrixelement at the intersection of the row identified in stage 430 and thecolumn identified in stage 450.

Referring to FIG. 4 in more detail, at stage 410, the client interface132 performs initialization functions to prepare for input selection.For example, the client interface 132 may request sensor recalibration,may present a selection matrix (e.g., a virtual keyboard) on apresentation screen (e.g., at the client device 120), and beginmonitoring selection sensors.

At stage 420, the client interface 132 receives notification of a signalfrom a sensor 140 and determines if the signal is sufficient to takeaction (see, e.g., method 200 illustrated in FIG. 2 ). If not (e.g., ifthe signal is too short), the client interface 132 returns toinitialization stage 410 to wait for another signal. This use of adiscrete signal from one or more sensors for each action start, and foreach action transition, allows a user 122 to rest between selections. Insome implementations, timeouts are configured to allow for more or lessrest as needed by a particular user 122. However, in someimplementations, a user 122 may be expected to sustain a signal duringan action, e.g., maintaining a selection signal while an event takesplace.

If an actionable signal is received at stage 420, then at stage 430 theclient interface 132 begins scanning rows of the selection matrix. Theclient interface 132 continues to scan rows of the selection matrixuntil signaled to stop. The last row scanned in stage 430 is identifiedas a selected row. For example, in FIG. 3C, a highlighted row 360 ofvirtual keys is shown selected.

At stage 440, the client interface 132 receives notification of a signalfrom a sensor 140 and determines if the signal is sufficient totransition to a next action. If not (e.g., if the signal is too short),the client interface 132 returns to initialization stage 410 to restartthe method 400 and wait for another signal. At stage 440, the clientinterface 132 may also determine to return to stage 410 if too much timepasses since starting stage 430.

If, at stage 440, an actionable signal has been received, then at stage450 the client interface 132 begins scanning columns of the selectionmatrix. The client interface 132 continues to scan columns of theselection matrix until signaled to stop. The last column scanned instage 450 is identified as a selected column. The scanning in stage 450is similar to stage 430. The objective is to identify an intersectionbetween a row and a column, which can then be selected. For example, toidentify a key in a virtual keyboard 350.

At stage 460, the client interface 132 receives notification of a signalfrom a sensor 140 and determines if the signal is sufficient totransition to a next action. If not (e.g., if the signal is too short),the client interface 132 returns to scanning rows at stage 430 andwaiting for another signal. At stage 460, the client interface 132 mayalso determine to return to stage 430 if too much time passes sincestarting stage 450.

If, at stage 460, an actionable signal has been received, then at stage470 the client interface 132 selects the matrix element at theintersection of the row identified in stage 430 and the columnidentified in stage 450. For example, a single virtual key may beidentified (see, e.g., the virtual keyboard 350 shown in FIG. 3B, inwhich a single key is highlighted). The selected key is then acted upon,e.g., entering a corresponding letter in a text entry field 340.

FIG. 5 is a flowchart for an example method 500 of operating a pointerusing sEMG sensors. The method 500 is similar to the method 400described in reference to FIG. 4 , but differs in that the selectionmatrix is replaced by Cartesian coordinates or lines on a display. Forexample, as shown in FIG. 3A, scan lines 310 and 320 can be manipulatedto intersect over a selectable element such as a graphical userinterface (GUI) icon 330. Like the description of FIG. 4 , thisdescription of FIG. 5 refers to a single sensor, but is applicable toone or more sensors, including groups or sets of sensors. Thisdescription allocates all tasks to the client interface 132; however, insome implementations, some or all of these tasks may be performed bysome other element in the sensor hub 130, on a sensor 140, or even atthe client device 120. The method 500 identifies a Y axis coordinatewith a vertical scan before identifying an X axis coordinate with ahorizontal scan. In some implementations, horizontal scans are performedbefore vertical scans. In some implementations, a first set of sensorsis used for vertical scans and a separate set of sensors is used forhorizontal scans; whichever set of sensors is triggered determineswhether a vertical or horizontal scan is performed. See, for example,FIG. 7 and the corresponding description below. These implementationvariations and others are within the scope of this disclosure.

In broad overview of FIG. 5 , at stage 510, the client interface 132performs initialization functions to prepare for pointer or mouse inputselection. At stage 520, the client interface 132 receives notificationof a signal from a sensor 140 and determines if the signal is sufficientto take action (see, e.g., method 200 illustrated in FIG. 2 ). If not(e.g., if the signal is too short), the client interface 132 returns toinitialization stage 510 to wait for another signal. Otherwise, if thesignal is actionable, then at stage 530 the client interface 132 startsa vertical scan to identify a Y axis coordinate. The client interface132 continues the vertical scan until signaled to stop. At stage 540,the client interface 132 receives notification of a signal from a sensor140 and determines if the signal is sufficient to transition to a nextaction. If not (e.g., if the signal is too short), the client interface132 returns to initialization stage 510 to restart the method 500 andwait for another signal. At stage 540, the client interface 132 may alsodetermine to return to stage 510 if too much time passes since startingstage 530. If, at stage 540, an actionable signal has been received,then at stage 550 the client interface 132 starts a horizontal scan toidentify an X axis coordinate. The client interface 132 continues thehorizontal scan until signaled to stop. At stage 560, the clientinterface 132 receives notification of a signal from a sensor 140 anddetermines if the signal is sufficient to transition to a next action.If not (e.g., if the signal is too short), the client interface 132returns to the vertical scan in stage 530 and waiting for anothersignal. At stage 560, the client interface 132 may also determine toreturn to stage 530 if too much time passes since starting stage 550.If, at stage 560, an actionable signal has been received, then at stage570 the client interface 132 identifies and selects the intersection ofthe vertical identified in stage 530 and the horizontal identified instage 550.

Referring to FIG. 5 in more detail, at stage 510, the client interface132 performs initialization functions to prepare for pointer or mouseinput selection. For example, like stage 410 in FIG. 4 , the clientinterface 132 may request sensor recalibration, may present a horizontalscan line 320 (or vertical scan line 310) in an initial position on apresentation screen (e.g., at the client device 120), and may beginmonitoring selection sensors. In some implementations, the clientinterface 132 switches between a mouse mode and a matrix mode (e.g., avirtual keyboard mode). In such implementations, the client interface132 enters mouse mode in stage 510.

At stage 520, the client interface 132 receives notification of a signalfrom a sensor 140 and determines if the signal is sufficient to takeaction (see, e.g., method 200 illustrated in FIG. 2 ). If not (e.g., ifthe signal is too short), the client interface 132 returns toinitialization stage 510 to wait for another signal. Otherwise, if thesignal is actionable, then the client interface 132 proceeds to stage530.

At stage 530 the client interface 132 performs a vertical scan toidentify a Y axis coordinate. The client interface 132 continues thevertical scan until signaled to stop. In some implementations, duringthe vertical scan, the client interface 132 causes a horizontal scanline 320 to be drawn on the screen and, as the vertical scan progresses,the line is moved vertically up (or down) the screen. In someimplementations, during the vertical scan, a value for the Y coordinatecorresponding to the horizontal scan line 320 is displayed on thescreen. In some implementations, the rate at which the horizontal scanline 320 is moved (or redrawn) can be configured. In someimplementations, a user 122 can signal for the progression to speed upor slow down.

At stage 540, the client interface 132 receives notification of a signalfrom a sensor 140 and determines if the signal is sufficient totransition to a next action. If not (e.g., if the signal is too short),the client interface 132 returns to initialization stage 510 to restartthe method 500 and wait for another signal. At stage 540, the clientinterface 132 may also determine to return to stage 510 if too much timepasses since starting stage 530. If, at stage 540, an actionable signalhas been received, then the client interface 132 proceeds to stage 550.

At stage 550 the client interface 132 performs a horizontal scan toidentify a X axis coordinate. The client interface 132 continues thehorizontal scan until signaled to stop. In some implementations, duringthe horizontal scan, the client interface 132 causes a vertical scanline 310 to be drawn on the screen and, as the horizontal scanprogresses, the line is moved horizontally across the screen. In someimplementations, during the horizontal scan, a value for the Xcoordinate corresponding to the vertical scan line 310 is displayed onthe screen. In some implementations, the rate at which the vertical scanline 310 is moved (or redrawn) can be configured. In someimplementations, a user 122 can signal for the progression to speed upor slow down.

At stage 560, the client interface 132 receives notification of a signalfrom a sensor 140 and determines if the signal is sufficient totransition to a next action. If not (e.g., if the signal is too short),the client interface 132 returns to the vertical scan in stage 530 andwaiting for another signal. At stage 560, the client interface 132 mayalso determine to return to stage 530 if too much time passes sincestarting stage 550. If, at stage 560, an actionable signal has beenreceived, then the client interface 132 proceeds to stage 570.

At stage 570 the client interface 132 identifies and selects theintersection of the Y axis identified in stage 530 and the X axisidentified in stage 550. For example, referring to FIG. 3A, the scanlines 310 and 320 intersect at an actionable icon 330. At stage 570, theclient interface 132 selects the icon 330 and thereby invokes an actionassociated with the icon 330. For example, the scan lines 310 and 320may intersect over an application icon, a menu icon, a virtual button,or some other selectable or actionable element. In some implementations,a user 122 can trigger one or more sensors in a first set of sensors tocause a first action at the identified intersection, and can trigger oneor more sensors in a second set of sensors to cause a second action atthe identified intersection. For example, sensors on a user's left sidemay be used to indicate a “left click,” and sensors on a user's rightside may be used to indicate a “right click.”

FIG. 6 is a flowchart for an example method 600 of selecting a menuoption using sEMG sensors. In broad overview of the method 600, at stage610, the client interface 132 presents a selectable menu option. Forexample, in some implementations, a row in a drop down menu is shownhighlighted. In some implementations, a radio button or check box isshown selected. At the method 600 progresses, different menu options maybe shown highlighted or selected. At stage 620, the client interface 132receives notification of a signal from a sensor 140 and determines ifthe signal is sufficient to transition to a next action, e.g., acceptingthe menu selection or moving to another menu option. If, at stage 620,the client interface 132 determines that the signal from the sensormeets the criteria for selecting the menu option, then at stage 630, theclient interface 132 invokes the selected menu option.

FIG. 7 is a flowchart for an example method 700 of differentiatingmultiple sEMG sensors. In some implementations, sensors 140 can beassigned to different selection groups. When the client interface 132receives a signal from one (or more) sensors in a set of sensorsassociated with a specific action, the client interface 132 sendscontrol messages to the client device 120 to invoke the specific action.For example, sensors placed on a user's 122 right arm may be used forone action, sensors placed on the user's 122 left arm may be used foranother action, sensors placed on the user's 122 legs may be used foranother action, and so on. Placing multiple sensors along nerve path,e.g., along the forearm, allows for multiple opportunities to detect anerve activation. In some implementations, the different sensors alongthe nerve path may detect slight signal elevations at slightly differentmoments, as the signal propagates down the arm. Analysis of the set ofsensors taken together allow for an action associated with the set ofsensors. For people with more control over nerve activation, this candouble or triple message throughput. For example, using a left armsignal for vertical motion and a right arm signal for horizontal motionwould allow the methods 400 and 500 to quickly reach a selection state.Referring to FIG. 7 in broad overview, at stage 710, the clientinterface 132 receives a signal from one set of multiple possible setsof sensors. At stage 720, the client interface 132 determines if thesignal was received from a sensor in a set associated with a specificaction. If so, then at stage 730, the client interface 132 causes thespecific action to be performed. Otherwise, at stage 740, a defaultaction (or no action) is performed.

A person can control multiple electronics at the same time and maintaincommunication seamlessly across environments. The disclosed systemsprovide users with multiple inputs for efficient on-screen keyboard andmouse control. A person could wear 1, 2, 3, 4, 6, or more sensorsanywhere on his or her body. For example, one sensor could activate aleft mouse click while another activates a right mouse click. Previoussolutions had typing speeds around 6 words per minute; the disclosedkeyboard input solutions have facilitated input roughly three timesfaster. Web browsing is likewise simplified and accelerated. In someimplementations, software executed on either the sensor hub 130 orclient device 120 can automatically map the configuration of the sensors140 and auto-calibrate for conditions such as movement, sweat, andpressure.

FIG. 8 is a block diagram illustrating a general architecture of acomputing system 101 suitable for use in some implementations describedherein The example computing system 101 includes one or more processors107 in communication, via a bus 105, with one or more network interfaces111 (in communication with a network 110), I/O interfaces 102 (forinteracting with a user or administrator), and memory 106. The processor107 incorporates, or is directly connected to, additional cache memory109. In some uses, additional components are in communication with thecomputing system 101 via a peripheral interface 103. In some uses, suchas in a server context, there is no I/O interface 102 or the I/Ointerface 102 is not used. In some uses, the I/O interface 102 supportsan input device 104 and/or an output device 108. In some uses, the inputdevice 104 and the output device 108 use the same hardware, for example,as in a touch screen. In some uses, the computing system 101 isstand-alone and does not interact with a network 110 and might not havea network interface 111.

The processor 107 may be any logic circuitry that processesinstructions, e.g., instructions fetched from the memory 106 or cache109. In many implementations, the processor 107 is a microprocessorunit. The processor 107 may be any processor capable of operating asdescribed herein. The processor 107 may be a single core or multi-coreprocessor. The processor 107 may be multiple processors. In someimplementations, the processor 107 is augmented with a co-processor,e.g., a math co-processor or a graphics co-processor.

The I/O interface 102 may support a wide variety of devices. Examples ofan input device 104 include a keyboard, mouse, touch or track pad,trackball, microphone, touch screen, or drawing tablet. Example of anoutput device 108 include a video display, touch screen, refreshableBraille display, speaker, inkjet printer, laser printer, or 3D printer.In some implementations, an input device 104 and/or output device 108may function as a peripheral device connected via a peripheral interface103.

A peripheral interface 103 supports connection of additional peripheraldevices to the computing system 101. The peripheral devices may beconnected physically, as in a universal serial bus (“USB”) device, orwirelessly, as in a BLUETOOTH™ device. Examples of peripherals includekeyboards, pointing devices, display devices, audio devices, hubs,printers, media reading devices, storage devices, hardware accelerators,sound processors, graphics processors, antennas, signal receivers,measurement devices, and data conversion devices. In some uses,peripherals include a network interface and connect with the computingsystem 101 via the network 110 and the network interface 111. Forexample, a printing device may be a network accessible printer.

The network 110 is any network, e.g., as shown and described above inreference to FIG. 1 . Examples of networks include a local area network(“LAN”), a wide area network (“WAN”), an inter-network (e.g., theInternet), and peer-to-peer networks (e.g., ad hoc peer-to-peernetworks). The network 110 may be composed of multiple connectedsub-networks and/or autonomous systems. Any type and/or form of datanetwork and/or communication network can be used for the network 110.

The memory 106 may each be implemented using one or more data storagedevices. The data storage devices may be any memory device suitable forstoring computer readable data. The data storage devices may include adevice with fixed storage or a device for reading removable storagemedia. Examples include all forms of non-volatile memory, media andmemory devices, semiconductor memory devices (e.g., EPROM, EEPROM,SDRAM, and flash memory devices), magnetic disks, magneto optical disks,and optical discs (e.g., CD ROM, DVD-ROM, or BLU-RAY discs). Exampleimplementations of suitable data storage devices include storage areanetworks (“SAN”), network attached storage (“NAS”), and redundantstorage arrays.

The cache 109 is a form of data storage device place on the same circuitstrata as the processor 107 or in close proximity thereto. In someimplementations, the cache 109 is a semiconductor memory device. Thecache 109 may be include multiple layers of cache, e.g., L1, L2, and L3,where the first layer is closest to the processor 107 (e.g., on chip),and each subsequent layer is slightly further removed. Generally, cache109 is a high-speed low-latency memory.

The computing system 101 can be any workstation, desktop computer,laptop or notebook computer, server, handheld computer, mobile telephoneor other portable telecommunication device, media playing device, agaming system, mobile computing device, or any other type and/or form ofcomputing, telecommunications or media device that is capable ofcommunication and that has sufficient processor power and memorycapacity to perform the operations described herein. In someimplementations, one or more devices are constructed to be similar tothe computing system 101 of FIG. 8 . In some implementations, multipledistinct devices interact to form, in the aggregate, a system similar tothe computing system 101 of FIG. 8 .

In some implementations, a server may be a virtual server, for example,a cloud-based server accessible via the network 110. A cloud-basedserver may be hosted by a third-party cloud service host. A server maybe made up of multiple computer systems 101 sharing a location ordistributed across multiple locations. The multiple computer systems 101forming a server may communicate using the network 110. In someimplementations, the multiple computer systems 101 forming a servercommunicate using a private network, e.g., a private backbone networkdistinct from a publicly-accessible network, or a virtual privatenetwork within a publicly-accessible network.

It should be understood that the systems and methods described above maybe provided as instructions in one or more computer programs recorded onor in one or more articles of manufacture, e.g., computer-readablemedia. The article of manufacture may be a floppy disk, a hard disk, aCD-ROM, a flash memory card, a PROM, a RAM, a ROM, or a magnetic tape.In general, the computer programs may be implemented in any programminglanguage, such as C, C++, C#, LISP, Perl, PROLOG, Python, Ruby, or inany byte code language such as JAVA. The software programs may be storedon or in one or more articles of manufacture as object code. The articleof manufacture stores this data in a non-transitory form.

While this specification contains many specific implementation details,these descriptions are of features specific to various particularimplementations and should not be construed as limiting. Certainfeatures described in the context of separate implementations can alsobe implemented in a unified combination. Additionally, many featuresdescribed in the context of a single implementation can also beimplemented separately or in various sub-combinations. Similarly, whileoperations are depicted in the figures in a particular order, thisshould not be understood as requiring that such operations be performedin the particular order shown or in sequential order, or that allillustrated operations be performed, to achieve desirable results. Incertain circumstances, multitasking and parallel processing may beadvantageous. Moreover, the separation of various system components inthe embodiments described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedin a single software product or packaged into multiple softwareproducts.

References to “or” may be construed as inclusive so that any termsdescribed using “or” may indicate any of a single, more than one, andall of the described terms. Likewise, references to “and/or” may beconstrued as an explicit use of the inclusive “or.” The labels “first,”“second,” “third,” an so forth are not necessarily meant to indicate anordering and are generally used merely as labels to distinguish betweenlike or similar items or elements.

Having described certain implementations and embodiments of methods andsystems, it will now become apparent to one of skill in the art thatother embodiments incorporating the concepts of the disclosure may beused. Therefore, the disclosure should not be limited to certainimplementations or embodiments, but rather should be limited only by thespirit and scope of the following claims.

Thus, some rudimentary solutions exist to help a person with AmyotrophicLateral Sclerosis (ALS) communicate with others, such as tracking eyegaze, but these solutions can be exhausting and difficult to use. Forexample, only about 10% of people with ALS can even use the eye gazemethod. As described herein, surface electromyography (sEMG) can be usedto control electrical equipment such as wheelchairs, remote controldoors, entertainment systems (e.g., radios and televisions), andcomputer systems. This new level of control can more effectively andefficiently facilitate communication than existing technologies and willimprove the over-all quality of life for persons with ALS.

Other implementations are also within the scope of the following claims.

For example, although we have used the example of the wrist in much ofour discussion, the technologies that we have described here can beapplied in devices that measure electrical potentials on any other partof the body, including the feet, arms, legs, torso, and brain, forexample.

Although we have discussed electrical signals appearing in muscles,bones, and nerves, it is possible that electrical signals may alsoappear in tendons, for example, because they may conduct or reflectelectrical signals of muscles, for example, in a frequency range similarto the frequency range of muscle electrical signals. Our descriptionrelated to electrical signals in muscles, nerves, and bones thereforemay also apply to tendons.

1. A system for biopotential-based gesture control, the systemcomprising: a plurality of electrode pairs, each electrode pair defininga channel configured to receive electrical signals at a respectiveposition on a portion of a body of a person; a motion sensor, the motionsensor being configured to generate a motion sensor output indicating amotion of the portion of the body of the person; a controller, thecontroller being configured to control processing of the electricalsignals received by the plurality of electrode pairs; a classifier, theclassifier being configured to analyze data derived from the pluralityof electrode pairs and the motion sensor and, based on this analysis,generate interpreted outputs indicating gestures performed by theperson; wherein the system is configured to: enter a monitoring state inwhich the system is configured to receive data from the plurality ofelectrode pairs and the motion sensor; receive a first set of data, thefirst set of data being derived from the electrical signals of one ormore of the plurality of electrode pairs and/or the motion sensoroutput; determine that the first set of data does not satisfy one ormore action criteria; in response to determining that the first set ofdata does not satisfy the one or more action criteria, return to themonitoring state without transmitting the first set of data to theclassifier; receive a second set of data, the second set of data beingderived from the electrical signals of one or more of the plurality ofelectrode pairs and/or the motion sensor output; determine that thesecond set of data satisfies the one or more action criteria; inresponse to determining that the second set of data satisfies the one ormore action criteria, transmit the second set of data to the classifier;using the classifier, analyze the second set of data to generate aninterpreted output indicating a gesture performed by the person; andbased on the interpreted output, generate a machine instruction.
 2. Thesystem of claim 1, wherein determining that the second set of datasatisfies the one or more action criteria comprises determining that anamplitude of one or more signals received by the plurality of electrodepairs exceeds a threshold.
 3. The system of claim 2, wherein thethreshold is determined based on electrical noise in an environment ofthe plurality of electrode pairs.
 4. The system of claim 1, whereindetermining that the second set of data satisfies the one or more actioncriteria comprises determining that a duration of one or more signalsreceived by the plurality of electrode pairs exceeds a threshold.
 5. Thesystem of claim 1, wherein determining that the second set of datasatisfies the one or more action criteria comprises determining that ameasured parameter of one or more signals received by the plurality ofelectrode pairs exceeds a threshold.
 6. The system of claim 1, wherein:the system is further configured to compare a first measurement from afirst electrode pair of the plurality of electrode pairs to a secondmeasurement from a second pair of the plurality of electrode pairs; anddetermining that the second set of data satisfies the one or more actioncriteria is based, at least in part, on the comparison between the firstmeasurement and the second measurement.
 7. The system of claim 6,wherein the system is configured to assess, based on the comparisonbetween the first measurement and the second measurement, that aphenomenon captured in both the first measurement and the secondmeasurement is a neuromuscular activation intended by the person.
 8. Thesystem of claim 7, wherein the comparison between the first measurementand the second measurement comprises determining that the phenomenonregistered at the first electrode pair earlier than at the secondelectrode pair.
 9. The system of claim 7, wherein the comparison betweenthe first measurement and the second measurement comprises determiningthat the phenomenon had a larger amplitude as measured at the firstelectrode pair than at the second electrode pair.
 10. The system ofclaim 1, wherein: the plurality of electrode pairs, the motion sensor,and the controller are disposed on a wearable device configured to beworn on a wrist of the person; and the classifier is disposed in asmartphone, the wearable device being configured to wirelessly transmitthe second data to the smartphone.
 11. A method for biopotential-basedgesture control, the method being performed using a system comprising aplurality of electrode pairs, a motion sensor, a controller, and aclassifier, the method comprising: entering a monitoring state in whichthe system is configured to receive data from the plurality of electrodepairs and the motion sensor, each electrode pair of the plurality ofelectrode pairs defining a channel configured to receive electricalsignals at a respective position on a portion of a body of a person, andthe motion sensor being configured to generate a motion sensor outputindicating a motion of the portion of the body of the person; receivinga first set of data, the first set of data being derived from theelectrical signals of one or more of the plurality of electrode pairsand/or the motion sensor output; using the controller, determining thatthe first set of data does not satisfy one or more action criteria, thecontroller being configured to control processing of the electricalsignals received by the plurality of electrode pairs; in response todetermining that the first set of data does not satisfy the one or moreaction criteria, returning to the monitoring state without transmittingthe first set of data to the classifier; receiving a second set of data,the second set of data being derived from the electrical signals of oneor more of the plurality of electrode pairs and/or the motion sensoroutput; using the controller, determining that the second set of datasatisfies the one or more action criteria; in response to determiningthat the second set of data satisfies the one or more action criteria,transmitting the second set of data to the classifier, the classifierbeing configured to analyze data derived from the plurality of electrodepairs and the motion sensor and, based on this analysis, generateinterpreted outputs indicating gestures performed by the person; usingthe classifier, analyze the second set of data to generate aninterpreted output indicating a gesture performed by the person; andbased on the interpreted output, generate a machine instruction.
 12. Themethod of claim 11, wherein determining that the second set of datasatisfies the one or more action criteria comprises determining that anamplitude of one or more signals received by the plurality of electrodepairs exceeds a threshold.
 13. The method of claim 12, wherein thethreshold is determined based on electrical noise in an environment ofthe plurality of electrode pairs.
 14. The method of claim 11, whereindetermining that the second set of data satisfies the one or more actioncriteria comprises determining that a duration of one or more signalsreceived by the plurality of electrode pairs exceeds a threshold. 15.The method of claim 11, wherein determining that the second set of datasatisfies the one or more action criteria comprises determining that ameasured parameter of one or more signals received by the plurality ofelectrode pairs exceeds a threshold.
 16. The method of claim 11,wherein: the method further comprises comparing a first measurement froma first electrode pair of the plurality of electrode pairs to a secondmeasurement from a second pair of the plurality of electrode pairs; anddetermining that the second set of data satisfies the one or more actioncriteria is based, at least in part, on the comparison between the firstmeasurement and the second measurement.
 17. The method of claim 16,wherein the method further comprises assessing, based on the comparisonbetween the first measurement and the second measurement, that aphenomenon captured in both the first measurement and the secondmeasurement is a neuromuscular activation intended by the person. 18.The method of claim 17, wherein the comparison between the firstmeasurement and the second measurement comprises determining that thephenomenon registered at the first electrode pair earlier than at thesecond electrode pair.
 19. The method of claim 17, wherein thecomparison between the first measurement and the second measurementcomprises determining that the phenomenon had a larger amplitude asmeasured at the first electrode pair than at the second electrode pair.20. The method of claim 11, wherein: the plurality of electrode pairs,the motion sensor, and the controller are disposed on a wearable deviceconfigured to be worn on a wrist of the person; and the classifier isdisposed in a smartphone, the wearable device being configured towirelessly transmit the second data to the smartphone.